[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:45 dist-test-slave-srgv krb5kdc[28671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:45 dist-test-slave-srgv krb5kdc[28671](info): set up 2 sockets
Aug 08 15:54:45 dist-test-slave-srgv krb5kdc[28671](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 15:54:46.316025 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:54:46 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:54:46.493247 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40541
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:46.825042 28684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:46.825336 28684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:46.825528 28684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:46.832831 28684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:46.832970 28684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:46.833065 28684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:46.833168 28684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:46.833241 28684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:46.843598 28684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:46.845153 28684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:46.846705 28684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:46.859354 28689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:46.859319 28690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:46.859393 28692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:46.860536 28684 server_base.cc:1034] running on GCE node
I20260808 15:54:46.861315 28684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:46.862727 28684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:46.863952 28684 hybrid_clock.cc:648] HybridClock initialized: now 1786204486863910 us; error 71 us; skew 500 ppm
I20260808 15:54:46.864302 28684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:46 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204486, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:46.872502 28684 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:54:46.874173 28684 webserver.cc:460] Webserver started at http://127.27.254.62:38859/ using document root <none> and password file <none>
I20260808 15:54:46.874923 28684 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:46.875073 28684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:46.875387 28684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:46.878856 28684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "07350067b0f54fc496a6d1a50343154d"
format_stamp: "Formatted at 2026-08-08 15:54:46 on dist-test-slave-srgv"
server_key: "7121f38ee50f5e496a7b1aae4b2f6905"
server_key_iv: "064bc0014e9c4f328a4aac5abd146082"
server_key_version: "encryptionkey@0"
I20260808 15:54:46.879716 28684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "07350067b0f54fc496a6d1a50343154d"
format_stamp: "Formatted at 2026-08-08 15:54:46 on dist-test-slave-srgv"
server_key: "7121f38ee50f5e496a7b1aae4b2f6905"
server_key_iv: "064bc0014e9c4f328a4aac5abd146082"
server_key_version: "encryptionkey@0"
I20260808 15:54:46.884791 28684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:54:46.888401 28699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:46.889811 28684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:46.889989 28684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "07350067b0f54fc496a6d1a50343154d"
format_stamp: "Formatted at 2026-08-08 15:54:46 on dist-test-slave-srgv"
server_key: "7121f38ee50f5e496a7b1aae4b2f6905"
server_key_iv: "064bc0014e9c4f328a4aac5abd146082"
server_key_version: "encryptionkey@0"
I20260808 15:54:46.890189 28684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:46.905364 28684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:46.910147 28684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:46.910439 28684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:46.929208 28684 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:40541
I20260808 15:54:46.929227 28750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:40541 every 8 connection(s)
I20260808 15:54:46.930693 28684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:46.934502 28751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:46.939860 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 28684
I20260808 15:54:46.940207 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:54:46.942718 28664 external_mini_cluster.cc:1428] Setting key 5b0bd9a4cf257463405130846105432f
I20260808 15:54:46.947698 28751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d: Bootstrap starting.
I20260808 15:54:46.951169 28751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:46.952229 28751 log.cc:824] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:46.954926 28751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d: No bootstrap required, opened a new log
I20260808 15:54:46.964202 28751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07350067b0f54fc496a6d1a50343154d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40541 } }
I20260808 15:54:46.964586 28751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:46.964694 28751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07350067b0f54fc496a6d1a50343154d, State: Initialized, Role: FOLLOWER
I20260808 15:54:46.965312 28751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07350067b0f54fc496a6d1a50343154d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40541 } }
I20260808 15:54:46.965642 28751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:46.965796 28751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:46.965982 28751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:46.968241 28751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07350067b0f54fc496a6d1a50343154d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40541 } }
I20260808 15:54:46.968746 28751 leader_election.cc:302] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07350067b0f54fc496a6d1a50343154d; no voters:
I20260808 15:54:46.969374 28751 leader_election.cc:288] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:46.969545 28756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:46.971011 28756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [term 1 LEADER]: Becoming Leader. State: Replica: 07350067b0f54fc496a6d1a50343154d, State: Running, Role: LEADER
I20260808 15:54:46.971616 28756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07350067b0f54fc496a6d1a50343154d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40541 } }
I20260808 15:54:46.971740 28751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:46.974895 28758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07350067b0f54fc496a6d1a50343154d. Latest consensus state: current_term: 1 leader_uuid: "07350067b0f54fc496a6d1a50343154d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07350067b0f54fc496a6d1a50343154d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40541 } } }
I20260808 15:54:46.974933 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07350067b0f54fc496a6d1a50343154d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07350067b0f54fc496a6d1a50343154d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40541 } } }
I20260808 15:54:46.975350 28758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:46.975350 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d [sys.catalog]: This master's current role is: LEADER
Aug 08 15:54:46 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:46.978039 28762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:46.984051 28762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:46.992388 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:46.947706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.27.254.62:40541)
0808 15:54:46.948527 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:46.948542 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:46.951762 (+ 3220us) server_negotiation.cc:407] Connection header received
0808 15:54:46.961827 (+ 10065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:46.961855 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:46.962377 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:46.962844 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:46.965788 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:46.967202 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:46.968968 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:46.970230 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:46.971395 (+ 1165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:46.979195 (+ 7800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:46.979211 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:46.979230 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:46.979292 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:46.983902 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:46.985248 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:46.985258 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:46.985265 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:46.985536 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:46.986212 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:46.986216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:46.986220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:46.987134 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:46.987519 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:46.990188 (+ 2669us) server_negotiation.cc:299] Negotiation successful
0808 15:54:46.990586 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":348,"threads_started":1}
I20260808 15:54:47.003396 28762 catalog_manager.cc:1347] Generated new cluster ID: 95ebffb4d5c94adf85fe31720906043c
I20260808 15:54:47.003539 28762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:47.023620 28762 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:47.025197 28762 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:47.034091 28762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07350067b0f54fc496a6d1a50343154d: Generated new TSK 0
I20260808 15:54:47.034935 28762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:54:47.088793 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40541
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:47.425366 28777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:47.425657 28777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:47.425825 28777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:47.433223 28777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:47.433341 28777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:47.433570 28777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:54:47.443605 28777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:47.444999 28777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:47.446421 28777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:47.459826 28783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:47.459911 28785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:47.459947 28782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:47.460664 28777 server_base.cc:1034] running on GCE node
I20260808 15:54:47.461239 28777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:47.462162 28777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:47.463387 28777 hybrid_clock.cc:648] HybridClock initialized: now 1786204487463342 us; error 39 us; skew 500 ppm
I20260808 15:54:47.463685 28777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:47 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:47.466730 28777 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:54:47.468057 28777 webserver.cc:460] Webserver started at http://127.27.254.1:43927/ using document root <none> and password file <none>
I20260808 15:54:47.468745 28777 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:47.468887 28777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:47.469185 28777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:47.472319 28777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "af3f8e43ae9a484580f4332a57ffb454"
format_stamp: "Formatted at 2026-08-08 15:54:47 on dist-test-slave-srgv"
server_key: "04c12faac44eefbbf7b2c9e8f83d298d"
server_key_iv: "782bf6f953f1ad14ded75c879c7c59e0"
server_key_version: "encryptionkey@0"
I20260808 15:54:47.473148 28777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "af3f8e43ae9a484580f4332a57ffb454"
format_stamp: "Formatted at 2026-08-08 15:54:47 on dist-test-slave-srgv"
server_key: "04c12faac44eefbbf7b2c9e8f83d298d"
server_key_iv: "782bf6f953f1ad14ded75c879c7c59e0"
server_key_version: "encryptionkey@0"
I20260808 15:54:47.478147 28777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:54:47.481465 28792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:47.482707 28777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:54:47.482872 28777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "af3f8e43ae9a484580f4332a57ffb454"
format_stamp: "Formatted at 2026-08-08 15:54:47 on dist-test-slave-srgv"
server_key: "04c12faac44eefbbf7b2c9e8f83d298d"
server_key_iv: "782bf6f953f1ad14ded75c879c7c59e0"
server_key_version: "encryptionkey@0"
I20260808 15:54:47.483064 28777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:47.498149 28777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:47.502355 28777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:47.502817 28777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:47.504210 28777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:54:47.506124 28777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:47.506292 28777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:47.506518 28777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:47.506676 28777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:47.545679 28777 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:43279
I20260808 15:54:47.545792 28904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:43279 every 8 connection(s)
I20260808 15:54:47.546970 28777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:54:47.556150 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 28777
I20260808 15:54:47.556377 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:54:47.556972 28664 external_mini_cluster.cc:1428] Setting key 2eeb0580ee64c591dd98e3c2d21703a7
Aug 08 15:54:47 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:47.566565 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:47.549494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:58655 (local address 127.27.254.62:40541)
0808 15:54:47.549952 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:47.549959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:47.551051 (+ 1092us) server_negotiation.cc:407] Connection header received
0808 15:54:47.552895 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:47.552901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:47.552994 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:47.553109 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:47.554634 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:47.555352 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:47.557088 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:47.557544 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:47.557678 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:47.561108 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:47.561113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:47.561117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:47.561146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:47.563655 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:47.564325 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:47.564331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:47.564334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:47.564457 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:47.564861 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:47.564868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:47.564871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:47.565136 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:47.565295 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:47.565995 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 15:54:47.566207 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260808 15:54:47.567516 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:47.550162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40541 (local address 127.27.254.1:58655)
0808 15:54:47.550897 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:47.550940 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:47.552625 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:47.553346 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:47.553354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:47.553909 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:47.554470 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:47.554488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:47.555517 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:47.555521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:47.556949 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:47.556961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:47.557766 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:47.557773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:47.558021 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:47.558705 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:47.558732 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:47.560886 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:47.563795 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:47.563804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:47.563820 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:47.564210 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:47.564559 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:47.564564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:47.564568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:47.564751 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:47.565260 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:47.565268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:47.565649 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 15:54:47.565917 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 15:54:47.566431 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":320,"threads_started":1}
I20260808 15:54:47.569067 28905 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40541
I20260808 15:54:47.569443 28905 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:47.570406 28905 heartbeater.cc:511] Master 127.27.254.62:40541 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:54:47.572827 28716 ts_manager.cc:194] Registered new tserver with Master: af3f8e43ae9a484580f4332a57ffb454 (127.27.254.1:43279)
I20260808 15:54:47.574383 28716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:58655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:54:47.594735 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40541
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:47.924494 28911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:47.924817 28911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:47.924981 28911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:47.932341 28911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:47.932482 28911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:47.932662 28911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:54:47.942368 28911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:47.943660 28911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:47.945482 28911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:47.960736 28917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:47.960809 28919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:47.960760 28916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:47.961472 28911 server_base.cc:1034] running on GCE node
I20260808 15:54:47.962047 28911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:47.963014 28911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:47.964205 28911 hybrid_clock.cc:648] HybridClock initialized: now 1786204487964163 us; error 38 us; skew 500 ppm
I20260808 15:54:47.964526 28911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:47 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:47.967566 28911 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:54:47.968879 28911 webserver.cc:460] Webserver started at http://127.27.254.2:35453/ using document root <none> and password file <none>
I20260808 15:54:47.969631 28911 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:47.969760 28911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:47.970062 28911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:47.972883 28911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "c973aaa7764f4cbc8f59c7661e763a2f"
format_stamp: "Formatted at 2026-08-08 15:54:47 on dist-test-slave-srgv"
server_key: "54f30f97c9d0cacea703a25c5a62496c"
server_key_iv: "35d624fcd4fdb738943474581542ae78"
server_key_version: "encryptionkey@0"
I20260808 15:54:47.973694 28911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "c973aaa7764f4cbc8f59c7661e763a2f"
format_stamp: "Formatted at 2026-08-08 15:54:47 on dist-test-slave-srgv"
server_key: "54f30f97c9d0cacea703a25c5a62496c"
server_key_iv: "35d624fcd4fdb738943474581542ae78"
server_key_version: "encryptionkey@0"
I20260808 15:54:47.978688 28911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:54:47.982168 28926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:47.983448 28911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:54:47.983625 28911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "c973aaa7764f4cbc8f59c7661e763a2f"
format_stamp: "Formatted at 2026-08-08 15:54:47 on dist-test-slave-srgv"
server_key: "54f30f97c9d0cacea703a25c5a62496c"
server_key_iv: "35d624fcd4fdb738943474581542ae78"
server_key_version: "encryptionkey@0"
I20260808 15:54:47.983811 28911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:47.997007 28911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:48.000713 28911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:48.000978 28911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:48.001928 28911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:54:48.003173 28911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:48.003273 28911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:48.003413 28911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:48.003486 28911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:48.036041 28911 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:41745
I20260808 15:54:48.036085 29038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:41745 every 8 connection(s)
I20260808 15:54:48.037271 28911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:54:48.040521 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 28911
I20260808 15:54:48.040731 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:54:48.041173 28664 external_mini_cluster.cc:1428] Setting key 7ed925bde3fae0e48d29887670486346
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
I20260808 15:54:48.058931 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.039691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:57117 (local address 127.27.254.62:40541)
0808 15:54:48.039901 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.039905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.041235 (+ 1330us) server_negotiation.cc:407] Connection header received
0808 15:54:48.043252 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.043258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.043337 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.043438 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.044899 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.046130 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.047639 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.048386 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.048532 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.052465 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.052470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.052473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.052503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.056062 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.056750 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.056755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.056760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.056930 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.057328 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.057334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.057338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.057719 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.057909 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.058490 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.058700 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 15:54:48.059926 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.040245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40541 (local address 127.27.254.2:57117)
0808 15:54:48.041046 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:48.041091 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:48.042950 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:48.043627 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:48.043634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:48.044174 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:48.044720 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:48.044736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.046265 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.046268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:48.047494 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:48.047505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.048643 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.048648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:48.048838 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.049553 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:48.049574 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:48.052250 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:48.056199 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:48.056206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:48.056217 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:48.056629 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:48.057040 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:48.057043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:48.057045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:48.057215 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:48.057867 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:48.057873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:48.058187 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 15:54:48.058411 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 15:54:48.058852 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":340,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
I20260808 15:54:48.061339 29039 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40541
I20260808 15:54:48.061700 29039 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:48.062628 29039 heartbeater.cc:511] Master 127.27.254.62:40541 requested a full tablet report, sending...
I20260808 15:54:48.064245 28716 ts_manager.cc:194] Registered new tserver with Master: c973aaa7764f4cbc8f59c7661e763a2f (127.27.254.2:41745)
I20260808 15:54:48.065021 28716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:57117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:54:48.082561 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40541
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:48.407599 29044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:48.407882 29044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:48.408046 29044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:48.415604 29044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:48.415720 29044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:48.415908 29044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:54:48.425580 29044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:48.426879 29044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:48.428234 29044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:48.441222 29049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:48.441252 29052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:48.441298 29050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:48.442397 29044 server_base.cc:1034] running on GCE node
I20260808 15:54:48.443007 29044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:48.443910 29044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:48.445176 29044 hybrid_clock.cc:648] HybridClock initialized: now 1786204488445183 us; error 101 us; skew 500 ppm
I20260808 15:54:48.445672 29044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:48.449946 29044 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:54:48.451356 29044 webserver.cc:460] Webserver started at http://127.27.254.3:37689/ using document root <none> and password file <none>
I20260808 15:54:48.452140 29044 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:48.452250 29044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:48.452513 29044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:48.455436 29044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "413c0f07b11d438e893ef41e54d2f4b7"
format_stamp: "Formatted at 2026-08-08 15:54:48 on dist-test-slave-srgv"
server_key: "9b2804134757952ad081a6561e532b1a"
server_key_iv: "a69ab79052d398edf67fc98dcd7245ab"
server_key_version: "encryptionkey@0"
I20260808 15:54:48.456301 29044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "413c0f07b11d438e893ef41e54d2f4b7"
format_stamp: "Formatted at 2026-08-08 15:54:48 on dist-test-slave-srgv"
server_key: "9b2804134757952ad081a6561e532b1a"
server_key_iv: "a69ab79052d398edf67fc98dcd7245ab"
server_key_version: "encryptionkey@0"
I20260808 15:54:48.461534 29044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260808 15:54:48.464969 29059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:48.466269 29044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:54:48.466439 29044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "413c0f07b11d438e893ef41e54d2f4b7"
format_stamp: "Formatted at 2026-08-08 15:54:48 on dist-test-slave-srgv"
server_key: "9b2804134757952ad081a6561e532b1a"
server_key_iv: "a69ab79052d398edf67fc98dcd7245ab"
server_key_version: "encryptionkey@0"
I20260808 15:54:48.466650 29044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:48.488878 29044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:48.493345 29044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:48.493667 29044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:48.494683 29044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:54:48.495994 29044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:48.496071 29044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:48.496157 29044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:48.496202 29044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:48.529472 29044 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:34305
I20260808 15:54:48.529523 29171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:34305 every 8 connection(s)
I20260808 15:54:48.530861 29044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:54:48.539027 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29044
I20260808 15:54:48.539234 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:54:48.539729 28664 external_mini_cluster.cc:1428] Setting key b1022e396d7dbf00faab8c7c34790130
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:48.551755 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.533499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:59031 (local address 127.27.254.62:40541)
0808 15:54:48.533683 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.533688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.534914 (+ 1226us) server_negotiation.cc:407] Connection header received
0808 15:54:48.536802 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.536816 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.536936 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.537049 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.538689 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.539539 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.541806 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.542500 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.542652 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.546199 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.546204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.546208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.546238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.548751 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.549494 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.549498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.549500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.549639 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.550062 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.550066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.550068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.550359 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.550519 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.551171 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.551403 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:54:48.552702 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.534024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40541 (local address 127.27.254.3:59031)
0808 15:54:48.534757 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:48.534795 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:48.536524 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:48.537304 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:48.537312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:48.537918 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:48.538494 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:48.538512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.539694 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.539699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:48.541665 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:48.541679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.542705 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.542710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:48.542907 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.543711 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:48.543744 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:48.545991 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:48.548889 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:48.548898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:48.548914 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:48.549335 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:48.549756 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:48.549761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:48.549764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:48.549944 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:48.550488 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:48.550496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:48.550819 (+ 323us) client_negotiation.cc:769] Sending connection context
0808 15:54:48.551064 (+ 245us) client_negotiation.cc:240] Negotiation successful
0808 15:54:48.551575 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":295,"threads_started":1}
I20260808 15:54:48.554023 29172 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40541
I20260808 15:54:48.554360 29172 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:48.555230 29172 heartbeater.cc:511] Master 127.27.254.62:40541 requested a full tablet report, sending...
I20260808 15:54:48.556725 28716 ts_manager.cc:194] Registered new tserver with Master: 413c0f07b11d438e893ef41e54d2f4b7 (127.27.254.3:34305)
I20260808 15:54:48.557578 28716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:59031
I20260808 15:54:48.568292 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:54:48.577225 28905 heartbeater.cc:503] Master 127.27.254.62:40541 was elected leader, sending a full tablet report...
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:54:48.612485 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.598027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.27.254.1:43279)
0808 15:54:48.598715 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.598723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.598751 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:54:48.598913 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.598920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.599143 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.599277 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.600655 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.601704 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.602916 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.603480 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.603570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.607320 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.607341 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.607355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.607391 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.610039 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.610662 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.610669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.610671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.610786 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.611112 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.611115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.611117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.611447 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.611669 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.611884 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.612076 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":369,"threads_started":1}
W20260808 15:54:48.613739 28820 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43618
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:54:48.655108 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.641841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.27.254.1:43279)
0808 15:54:48.642077 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.642085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.642360 (+ 275us) server_negotiation.cc:407] Connection header received
0808 15:54:48.642568 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.642572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.642635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.642749 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.644104 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.644794 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.646061 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.646495 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.646623 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.650547 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.650552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.650555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.650588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.653038 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.653655 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.653658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.653660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.653797 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.654100 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.654103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.654105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.654334 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.654500 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.654735 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.654887 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:54:48.657263 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 28777
I20260808 15:54:48.669991 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.658151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.27.254.1:43279)
0808 15:54:48.658320 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.658324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.658760 (+ 436us) server_negotiation.cc:407] Connection header received
0808 15:54:48.658868 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.658871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.658920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.658995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.660114 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.660781 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.661944 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.662524 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.662648 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.665587 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.665593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.665597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.665633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.667902 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.668483 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.668487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.668489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.668615 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.668883 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.668886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.668888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.669093 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.669249 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.669524 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.669666 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:54:48.670818 28900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:48.753875 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 28777
I20260808 15:54:48.770798 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 28911
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:54:48.786267 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.771627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.27.254.2:41745)
0808 15:54:48.772196 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.772204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.772229 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:54:48.772288 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.772294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.772513 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.772651 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.773888 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.774871 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.776019 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.776550 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.776644 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.780696 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.780708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.780723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.780756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.783699 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.784292 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.784299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.784301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.784423 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.784775 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.784778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.784780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.785107 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.785320 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.785654 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.785862 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":306,"threads_started":1}
I20260808 15:54:48.787489 29034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:48.868193 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 28911
I20260808 15:54:48.883945 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29044
Aug 08 15:54:48 dist-test-slave-srgv krb5kdc[28671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:54:48.899384 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:48.884907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.27.254.3:34305)
0808 15:54:48.885561 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:48.885568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:48.885599 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:54:48.885677 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:48.885683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:48.885892 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:48.886014 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:48.887193 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.888166 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.889456 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:48.890029 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:48.890137 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:48.894145 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:48.894156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:48.894170 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:48.894203 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:48.896995 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.897663 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.897670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.897673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.897787 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:48.898080 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:48.898082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:48.898084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:48.898387 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:48.898612 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:48.898806 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 15:54:48.899002 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":297,"threads_started":1}
I20260808 15:54:48.900600 29167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:48.983958 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29044
I20260808 15:54:49.001562 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 28684
I20260808 15:54:49.002753 28746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:49.104491 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 28684
2026-08-08T15:54:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:49 dist-test-slave-srgv krb5kdc[29193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:49 dist-test-slave-srgv krb5kdc[29193](info): set up 2 sockets
Aug 08 15:54:49 dist-test-slave-srgv krb5kdc[29193](info): commencing operation
krb5kdc: starting...
W20260808 15:54:50.183308 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:54:50 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:54:50.364606 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37873
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:50.687150 29206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:50.687465 29206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:50.687551 29206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:50.694947 29206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:50.695063 29206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:50.695111 29206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:50.695159 29206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:50.695205 29206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:50.705591 29206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:50.707003 29206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:50.708739 29206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:50.720494 29214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:50.720561 29212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:50.720772 29211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:50.721649 29206 server_base.cc:1034] running on GCE node
I20260808 15:54:50.722400 29206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:50.723790 29206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:50.725028 29206 hybrid_clock.cc:648] HybridClock initialized: now 1786204490724976 us; error 56 us; skew 500 ppm
I20260808 15:54:50.725373 29206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:50 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204490, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:50.728358 29206 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:54:50.729851 29206 webserver.cc:460] Webserver started at http://127.27.254.62:41481/ using document root <none> and password file <none>
I20260808 15:54:50.730588 29206 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:50.730737 29206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:50.731060 29206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:50.734222 29206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "e1ad8cdd268d459a8372f017a6b64a48"
format_stamp: "Formatted at 2026-08-08 15:54:50 on dist-test-slave-srgv"
server_key: "7f188d3e0463f75dfb0e39752e2e8c2a"
server_key_iv: "cfda6fe20b50092e8c87d43a87ca48d2"
server_key_version: "encryptionkey@0"
I20260808 15:54:50.735010 29206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "e1ad8cdd268d459a8372f017a6b64a48"
format_stamp: "Formatted at 2026-08-08 15:54:50 on dist-test-slave-srgv"
server_key: "7f188d3e0463f75dfb0e39752e2e8c2a"
server_key_iv: "cfda6fe20b50092e8c87d43a87ca48d2"
server_key_version: "encryptionkey@0"
I20260808 15:54:50.740314 29206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:50.743830 29221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:50.745172 29206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:54:50.745366 29206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "e1ad8cdd268d459a8372f017a6b64a48"
format_stamp: "Formatted at 2026-08-08 15:54:50 on dist-test-slave-srgv"
server_key: "7f188d3e0463f75dfb0e39752e2e8c2a"
server_key_iv: "cfda6fe20b50092e8c87d43a87ca48d2"
server_key_version: "encryptionkey@0"
I20260808 15:54:50.745620 29206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:50.760392 29206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:50.764565 29206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:50.764878 29206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:50.783308 29206 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:37873
I20260808 15:54:50.783344 29272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:37873 every 8 connection(s)
I20260808 15:54:50.784901 29206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:50.789073 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:50.791961 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29206
I20260808 15:54:50.792207 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:54:50.793041 28664 external_mini_cluster.cc:1428] Setting key 5532a7142e49dd77d124135f0404a600
I20260808 15:54:50.802294 29273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48: Bootstrap starting.
I20260808 15:54:50.806245 29273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:50.807655 29273 log.cc:824] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:54:50 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:50.811374 29273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48: No bootstrap required, opened a new log
I20260808 15:54:50.818691 29276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:50.795285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.27.254.62:37873)
0808 15:54:50.796173 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:50.796185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:50.796229 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:54:50.797997 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:50.798021 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:50.798436 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:50.798776 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:50.800586 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:50.801985 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:50.803720 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:50.804429 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:50.804561 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:50.809684 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:50.809696 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:50.809724 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:50.809768 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:50.813797 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:50.814589 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:50.814599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:50.814608 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:50.814848 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:50.815213 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:50.815218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:50.815221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:50.815997 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:50.816283 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:50.816722 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 15:54:50.817222 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
I20260808 15:54:50.821938 29273 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ad8cdd268d459a8372f017a6b64a48" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37873 } }
I20260808 15:54:50.822270 29273 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:50.822345 29273 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ad8cdd268d459a8372f017a6b64a48, State: Initialized, Role: FOLLOWER
I20260808 15:54:50.822810 29273 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ad8cdd268d459a8372f017a6b64a48" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37873 } }
I20260808 15:54:50.823009 29273 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:50.823102 29273 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:50.823227 29273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:50.825299 29273 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ad8cdd268d459a8372f017a6b64a48" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37873 } }
I20260808 15:54:50.825946 29273 leader_election.cc:302] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1ad8cdd268d459a8372f017a6b64a48; no voters:
I20260808 15:54:50.826656 29273 leader_election.cc:288] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:50.826804 29278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:50.827487 29278 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [term 1 LEADER]: Becoming Leader. State: Replica: e1ad8cdd268d459a8372f017a6b64a48, State: Running, Role: LEADER
I20260808 15:54:50.828042 29278 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ad8cdd268d459a8372f017a6b64a48" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37873 } }
I20260808 15:54:50.828753 29273 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:50.830824 29280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ad8cdd268d459a8372f017a6b64a48. Latest consensus state: current_term: 1 leader_uuid: "e1ad8cdd268d459a8372f017a6b64a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ad8cdd268d459a8372f017a6b64a48" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37873 } } }
I20260808 15:54:50.831293 29280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:50.835122 29285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:50.835047 29279 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1ad8cdd268d459a8372f017a6b64a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ad8cdd268d459a8372f017a6b64a48" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37873 } } }
I20260808 15:54:50.835354 29279 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:50.841509 29285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:50.850592 29285 catalog_manager.cc:1347] Generated new cluster ID: 88621901129e49b0974b155e9452261e
I20260808 15:54:50.850725 29285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:50.863488 29285 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:50.864830 29285 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:50.874444 29285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1ad8cdd268d459a8372f017a6b64a48: Generated new TSK 0
I20260808 15:54:50.875476 29285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:54:50.929741 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37873
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:51.252223 29299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:51.252504 29299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:51.252667 29299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:51.260082 29299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:51.260203 29299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:51.260378 29299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:54:51.270292 29299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:51.271649 29299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:51.272976 29299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:51.286183 29304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:51.286257 29305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:51.286228 29307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:51.287389 29299 server_base.cc:1034] running on GCE node
I20260808 15:54:51.287935 29299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:51.288825 29299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:51.290015 29299 hybrid_clock.cc:648] HybridClock initialized: now 1786204491289957 us; error 50 us; skew 500 ppm
I20260808 15:54:51.290341 29299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:51 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:51.293356 29299 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:54:51.294872 29299 webserver.cc:460] Webserver started at http://127.27.254.1:40891/ using document root <none> and password file <none>
I20260808 15:54:51.295552 29299 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:51.295689 29299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:51.295992 29299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:51.298918 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "aafaecccab5a4f9ba061a99c5454dbff"
format_stamp: "Formatted at 2026-08-08 15:54:51 on dist-test-slave-srgv"
server_key: "2c777ddf4af8be516aa4265469a95f10"
server_key_iv: "f654a2339e2ce0f8ac14e7e6dc2fb81f"
server_key_version: "encryptionkey@0"
I20260808 15:54:51.299693 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "aafaecccab5a4f9ba061a99c5454dbff"
format_stamp: "Formatted at 2026-08-08 15:54:51 on dist-test-slave-srgv"
server_key: "2c777ddf4af8be516aa4265469a95f10"
server_key_iv: "f654a2339e2ce0f8ac14e7e6dc2fb81f"
server_key_version: "encryptionkey@0"
I20260808 15:54:51.304718 29299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:54:51.308190 29314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:51.309547 29299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:51.309715 29299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "aafaecccab5a4f9ba061a99c5454dbff"
format_stamp: "Formatted at 2026-08-08 15:54:51 on dist-test-slave-srgv"
server_key: "2c777ddf4af8be516aa4265469a95f10"
server_key_iv: "f654a2339e2ce0f8ac14e7e6dc2fb81f"
server_key_version: "encryptionkey@0"
I20260808 15:54:51.309922 29299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:51.339491 29299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:51.343791 29299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:51.344084 29299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:51.345064 29299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:54:51.346436 29299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:51.346534 29299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:51.346669 29299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:51.346746 29299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:51.379962 29299 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:45559
I20260808 15:54:51.380012 29426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:45559 every 8 connection(s)
I20260808 15:54:51.381258 29299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:54:51.387989 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29299
I20260808 15:54:51.388211 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:54:51.388827 28664 external_mini_cluster.cc:1428] Setting key 065d57f560d2947b408e0c7e4383753a
Aug 08 15:54:51 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:51.403805 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:51.383877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:46697 (local address 127.27.254.62:37873)
0808 15:54:51.384315 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:51.384320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:51.385213 (+ 893us) server_negotiation.cc:407] Connection header received
0808 15:54:51.387131 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:51.387137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:51.387244 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:51.387363 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:51.389009 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.390022 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.391931 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.392726 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.392888 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:51.396638 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:51.396647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:51.396655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:51.396701 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:51.400280 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:51.400993 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:51.401002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:51.401008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:51.401166 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:51.401655 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:51.401663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:51.401669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:51.402066 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:51.402244 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:51.403002 (+ 758us) server_negotiation.cc:299] Negotiation successful
0808 15:54:51.403288 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":245,"threads_started":1}
I20260808 15:54:51.404366 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:51.384371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37873 (local address 127.27.254.1:46697)
0808 15:54:51.385046 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:51.385085 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:51.386851 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:51.387606 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:51.387621 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:51.388193 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:51.388795 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:51.388814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.390191 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.390196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:51.391767 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:51.391781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.392959 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.392965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:51.393170 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:51.393943 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:51.393965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:51.396406 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:51.400427 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:51.400434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:51.400447 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:51.400865 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:51.401290 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:51.401293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:51.401295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:51.401545 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:51.402185 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:51.402190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:51.402603 (+ 413us) client_negotiation.cc:769] Sending connection context
0808 15:54:51.402828 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 15:54:51.403214 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":289,"threads_started":1}
I20260808 15:54:51.405913 29427 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37873
I20260808 15:54:51.406296 29427 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:54:51.407234 29427 heartbeater.cc:511] Master 127.27.254.62:37873 requested a full tablet report, sending...
I20260808 15:54:51.409320 29238 ts_manager.cc:194] Registered new tserver with Master: aafaecccab5a4f9ba061a99c5454dbff (127.27.254.1:45559)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
I20260808 15:54:51.411584 29238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:46697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:54:51.430995 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37873
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:51.767165 29433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:51.767447 29433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:51.767606 29433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:51.774914 29433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:51.775025 29433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:51.775238 29433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:54:51.786676 29433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:51.788169 29433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:51.789862 29433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:51.804479 29439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:51.804616 29441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:51.805326 29438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:51.805626 29433 server_base.cc:1034] running on GCE node
I20260808 15:54:51.806298 29433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:51.807296 29433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:51.808493 29433 hybrid_clock.cc:648] HybridClock initialized: now 1786204491808429 us; error 52 us; skew 500 ppm
I20260808 15:54:51.808776 29433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:51 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:51.811878 29433 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:54:51.813585 29433 webserver.cc:460] Webserver started at http://127.27.254.2:38849/ using document root <none> and password file <none>
I20260808 15:54:51.814323 29433 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:51.814457 29433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:51.814761 29433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:51.817677 29433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "3f18532e4c804a90b5b5f99943f757d6"
format_stamp: "Formatted at 2026-08-08 15:54:51 on dist-test-slave-srgv"
server_key: "9c873400d124201f5350f975bd664763"
server_key_iv: "ce1ee8c09d876dd4c31c8847291baf4d"
server_key_version: "encryptionkey@0"
I20260808 15:54:51.818445 29433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "3f18532e4c804a90b5b5f99943f757d6"
format_stamp: "Formatted at 2026-08-08 15:54:51 on dist-test-slave-srgv"
server_key: "9c873400d124201f5350f975bd664763"
server_key_iv: "ce1ee8c09d876dd4c31c8847291baf4d"
server_key_version: "encryptionkey@0"
I20260808 15:54:51.823338 29433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:54:51.826853 29448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:51.828089 29433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:54:51.828264 29433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "3f18532e4c804a90b5b5f99943f757d6"
format_stamp: "Formatted at 2026-08-08 15:54:51 on dist-test-slave-srgv"
server_key: "9c873400d124201f5350f975bd664763"
server_key_iv: "ce1ee8c09d876dd4c31c8847291baf4d"
server_key_version: "encryptionkey@0"
I20260808 15:54:51.828472 29433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:51.847522 29433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:51.852037 29433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:51.852339 29433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:51.853282 29433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:54:51.854667 29433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:51.854770 29433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:51.854918 29433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:51.854986 29433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:51.887356 29433 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:46621
I20260808 15:54:51.887396 29560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:46621 every 8 connection(s)
I20260808 15:54:51.888695 29433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:54:51.898262 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29433
I20260808 15:54:51.898463 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:54:51.899051 28664 external_mini_cluster.cc:1428] Setting key b6ad1e2afb0e0a35797ad35f974c6d49
Aug 08 15:54:51 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:51.915679 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:51.891168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:41251 (local address 127.27.254.62:37873)
0808 15:54:51.891341 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:51.891346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:51.892503 (+ 1157us) server_negotiation.cc:407] Connection header received
0808 15:54:51.894333 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:51.894338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:51.894405 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:51.894485 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:51.895953 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.896650 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.898207 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.898888 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.899017 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:51.908482 (+ 9465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:51.908492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:51.908500 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:51.908546 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:51.912279 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:51.913048 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:51.913055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:51.913061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:51.913234 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:51.913732 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:51.913738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:51.913743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:51.914138 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:51.914306 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:51.915006 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 15:54:51.915270 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
I20260808 15:54:51.916388 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:51.891680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37873 (local address 127.27.254.2:41251)
0808 15:54:51.892347 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:51.892383 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:51.894055 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:51.894681 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:51.894689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:51.895182 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:51.895797 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:51.895817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.896783 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.896786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:51.898064 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:51.898076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:51.899086 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:51.899091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:51.899393 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:51.905512 (+ 6119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:51.905557 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:51.908216 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:51.912440 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:51.912447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:51.912462 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:51.912904 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:51.913351 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:51.913354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:51.913356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:51.913601 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:51.914272 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:51.914278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:51.914622 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 15:54:51.914853 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 15:54:51.915254 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":260,"threads_started":1}
I20260808 15:54:51.917943 29561 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
I20260808 15:54:51.918286 29561 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:51.919020 29561 heartbeater.cc:511] Master 127.27.254.62:37873 requested a full tablet report, sending...
I20260808 15:54:51.920603 29238 ts_manager.cc:194] Registered new tserver with Master: 3f18532e4c804a90b5b5f99943f757d6 (127.27.254.2:46621)
I20260808 15:54:51.921923 29238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:41251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:54:51.940735 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37873
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:52.267604 29566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:52.267941 29566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:52.268113 29566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:52.275614 29566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:52.275771 29566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:52.275966 29566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:54:52.285892 29566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37873
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:52.287286 29566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:52.288594 29566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:52.301555 29571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:52.301668 29572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:52.301666 29574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:52.302665 29566 server_base.cc:1034] running on GCE node
I20260808 15:54:52.303264 29566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:52.304186 29566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:52.305382 29566 hybrid_clock.cc:648] HybridClock initialized: now 1786204492305345 us; error 42 us; skew 500 ppm
I20260808 15:54:52.305735 29566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:52.308760 29566 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:54:52.310328 29566 webserver.cc:460] Webserver started at http://127.27.254.3:44151/ using document root <none> and password file <none>
I20260808 15:54:52.311021 29566 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:52.311149 29566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:52.311474 29566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:52.314738 29566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "6558e9afc70140278c9c1eee4a53cfa9"
format_stamp: "Formatted at 2026-08-08 15:54:52 on dist-test-slave-srgv"
server_key: "c474aec3145487879d05e70d3e5f13f6"
server_key_iv: "c5c8d68038eb57fd4bf9a379f08a26ef"
server_key_version: "encryptionkey@0"
I20260808 15:54:52.315546 29566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "6558e9afc70140278c9c1eee4a53cfa9"
format_stamp: "Formatted at 2026-08-08 15:54:52 on dist-test-slave-srgv"
server_key: "c474aec3145487879d05e70d3e5f13f6"
server_key_iv: "c5c8d68038eb57fd4bf9a379f08a26ef"
server_key_version: "encryptionkey@0"
I20260808 15:54:52.320868 29566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:52.324371 29581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:52.325732 29566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:52.325933 29566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "6558e9afc70140278c9c1eee4a53cfa9"
format_stamp: "Formatted at 2026-08-08 15:54:52 on dist-test-slave-srgv"
server_key: "c474aec3145487879d05e70d3e5f13f6"
server_key_iv: "c5c8d68038eb57fd4bf9a379f08a26ef"
server_key_version: "encryptionkey@0"
I20260808 15:54:52.326179 29566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:52.343093 29566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:52.347142 29566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:52.347429 29566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:52.348376 29566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:54:52.349884 29566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:52.349977 29566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:52.350135 29566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:52.350204 29566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:52.384403 29566 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:41477
I20260808 15:54:52.384406 29693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:41477 every 8 connection(s)
I20260808 15:54:52.385772 29566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:54:52.388616 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29566
I20260808 15:54:52.388818 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:54:52.389235 28664 external_mini_cluster.cc:1428] Setting key ee5e84e93e7eadadb72fcd27147539dc
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:52.405740 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.388351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:39917 (local address 127.27.254.62:37873)
0808 15:54:52.388565 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.388571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.389966 (+ 1395us) server_negotiation.cc:407] Connection header received
0808 15:54:52.392165 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.392171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.392256 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.392369 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.393900 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.394638 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.396147 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.396647 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.396783 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.400233 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.400238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.400241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.400273 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.402746 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.403463 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.403468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.403472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.403610 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.404043 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.404048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.404052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.404315 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.404468 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.405146 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.405362 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:54:52.406667 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.388934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37873 (local address 127.27.254.3:39917)
0808 15:54:52.389791 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:52.389844 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:52.391921 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:52.392604 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:52.392612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:52.393114 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:52.393743 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.393760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.394799 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.394804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.396005 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.396016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.396895 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.396902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.397191 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.397903 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:52.397924 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:52.400036 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:52.402889 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:52.402897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:52.402914 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:52.403336 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:52.403720 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:52.403728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:52.403733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:52.403926 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:52.404442 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:52.404450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:52.404786 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 15:54:52.405042 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 15:54:52.405589 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":350,"threads_started":1}
I20260808 15:54:52.407971 29694 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37873
I20260808 15:54:52.408267 29694 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:52.408965 29694 heartbeater.cc:511] Master 127.27.254.62:37873 requested a full tablet report, sending...
I20260808 15:54:52.410391 29238 ts_manager.cc:194] Registered new tserver with Master: 6558e9afc70140278c9c1eee4a53cfa9 (127.27.254.3:41477)
I20260808 15:54:52.411195 29238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:39917
I20260808 15:54:52.415153 29427 heartbeater.cc:503] Master 127.27.254.62:37873 was elected leader, sending a full tablet report...
I20260808 15:54:52.416296 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:54:52.434938 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.422585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.27.254.62:37873)
0808 15:54:52.422751 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.422755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.423280 (+ 525us) server_negotiation.cc:407] Connection header received
0808 15:54:52.423423 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.423427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.423484 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.423564 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.424978 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.425747 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.426967 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.427360 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.427447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.429836 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.429840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.429843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.429871 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.432537 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.433161 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.433164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.433167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.433289 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.433713 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.433716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.433718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.434064 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.434244 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.434572 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.434722 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:54:52.441201 29237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:54:52.447890 29237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:54:52.480322 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.459927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.27.254.3:41477)
0808 15:54:52.471505 (+ 11578us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.471518 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.472513 (+ 995us) server_negotiation.cc:407] Connection header received
0808 15:54:52.472663 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.472671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.472985 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.473189 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:52.473896 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.475205 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.477638 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.479012 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.479155 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.479420 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.479801 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.479959 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11279,"thread_start_us":11568,"threads_started":1}
I20260808 15:54:52.480353 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.459616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:41477 (local address 127.0.0.1:57266)
0808 15:54:52.472212 (+ 12596us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:52.472235 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:52.472368 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:52.473202 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:52.473209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:52.473241 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:52.473693 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.473705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.475375 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.475381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.477448 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.477464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.479364 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.479372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.479554 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.479591 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:54:52.479732 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 15:54:52.479925 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12452,"mutex_wait_us":31,"thread_start_us":17734,"threads_started":1}
I20260808 15:54:52.482009 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.459748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.27.254.2:46621)
0808 15:54:52.472470 (+ 12722us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.472753 (+ 283us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.472793 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:54:52.472907 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.472918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.473195 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.474304 (+ 1109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:52.474571 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.476401 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.479388 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.480890 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.481011 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.481215 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.481495 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.481673 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12502,"thread_start_us":12751,"threads_started":1}
I20260808 15:54:52.482034 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.459551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:46621 (local address 127.0.0.1:48934)
0808 15:54:52.460296 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:52.460332 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:52.460517 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:52.473493 (+ 12976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:52.473501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:52.473537 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:52.474068 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.474083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.476561 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.476567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.479169 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.479182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.481074 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.481080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.481259 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.481300 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 15:54:52.481437 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 15:54:52.481580 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":1,"thread_start_us":1090,"threads_started":1}
I20260808 15:54:52.484699 29629 tablet_service.cc:1467] Processing CreateTablet for tablet 1519f8a41ca34bedad6782a45e8d1224 (DEFAULT_TABLE table=test-table [id=2c1a9be32bd840f798a4439c5fe2b1df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:52.485627 29496 tablet_service.cc:1467] Processing CreateTablet for tablet 1519f8a41ca34bedad6782a45e8d1224 (DEFAULT_TABLE table=test-table [id=2c1a9be32bd840f798a4439c5fe2b1df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:52.486303 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1519f8a41ca34bedad6782a45e8d1224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:52.486699 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.461754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45559 (local address 127.0.0.1:48014)
0808 15:54:52.477669 (+ 15915us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:52.477692 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:52.477835 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:52.478682 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:52.478687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:52.478708 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:52.479183 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.479192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.481259 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.481264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.483792 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.483806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.485909 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.485916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.486078 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.486118 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:54:52.486219 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:54:52.486363 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15772,"mutex_wait_us":1,"thread_start_us":10174,"threads_started":1}
I20260808 15:54:52.486744 29496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1519f8a41ca34bedad6782a45e8d1224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:52.487452 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.462132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.27.254.1:45559)
0808 15:54:52.477111 (+ 14979us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.477122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.477831 (+ 709us) server_negotiation.cc:407] Connection header received
0808 15:54:52.477967 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.477975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.478246 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.478450 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:52.479402 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.481082 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.483968 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.485362 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.485522 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.485760 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.486924 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.487099 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14766,"thread_start_us":14586,"threads_started":1}
I20260808 15:54:52.491250 29362 tablet_service.cc:1467] Processing CreateTablet for tablet 1519f8a41ca34bedad6782a45e8d1224 (DEFAULT_TABLE table=test-table [id=2c1a9be32bd840f798a4439c5fe2b1df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:52.492678 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1519f8a41ca34bedad6782a45e8d1224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:52.498093 29712 tablet_bootstrap.cc:492] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: Bootstrap starting.
I20260808 15:54:52.501046 29713 tablet_bootstrap.cc:492] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: Bootstrap starting.
I20260808 15:54:52.502297 29712 tablet_bootstrap.cc:654] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:52.503619 29712 log.cc:824] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:52.504562 29713 tablet_bootstrap.cc:654] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:52.505728 29713 log.cc:824] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:52.506868 29712 tablet_bootstrap.cc:492] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: No bootstrap required, opened a new log
I20260808 15:54:52.507143 29712 ts_tablet_manager.cc:1397] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 15:54:52.507153 29716 tablet_bootstrap.cc:492] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: Bootstrap starting.
I20260808 15:54:52.508541 29713 tablet_bootstrap.cc:492] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: No bootstrap required, opened a new log
I20260808 15:54:52.508719 29713 ts_tablet_manager.cc:1397] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 15:54:52.510542 29716 tablet_bootstrap.cc:654] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:52.511739 29716 log.cc:824] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:52.514866 29716 tablet_bootstrap.cc:492] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: No bootstrap required, opened a new log
I20260808 15:54:52.515121 29716 ts_tablet_manager.cc:1397] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:54:52.516906 29713 raft_consensus.cc:348] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.517292 29713 raft_consensus.cc:374] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:52.517390 29713 raft_consensus.cc:729] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f18532e4c804a90b5b5f99943f757d6, State: Initialized, Role: FOLLOWER
I20260808 15:54:52.517577 29712 raft_consensus.cc:348] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.518050 29712 raft_consensus.cc:374] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:52.518154 29712 raft_consensus.cc:729] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6558e9afc70140278c9c1eee4a53cfa9, State: Initialized, Role: FOLLOWER
I20260808 15:54:52.518049 29713 consensus_queue.cc:260] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.518750 29712 consensus_queue.cc:260] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.519383 29561 heartbeater.cc:503] Master 127.27.254.62:37873 was elected leader, sending a full tablet report...
I20260808 15:54:52.519948 29713 ts_tablet_manager.cc:1428] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260808 15:54:52.519927 29694 heartbeater.cc:503] Master 127.27.254.62:37873 was elected leader, sending a full tablet report...
I20260808 15:54:52.521838 29712 ts_tablet_manager.cc:1428] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260808 15:54:52.526379 29716 raft_consensus.cc:348] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.526775 29716 raft_consensus.cc:374] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:52.526882 29716 raft_consensus.cc:729] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aafaecccab5a4f9ba061a99c5454dbff, State: Initialized, Role: FOLLOWER
I20260808 15:54:52.527393 29716 consensus_queue.cc:260] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.528970 29716 ts_tablet_manager.cc:1428] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260808 15:54:52.562321 29719 raft_consensus.cc:482] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:54:52.562640 29719 raft_consensus.cc:504] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.564312 29719 leader_election.cc:288] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f18532e4c804a90b5b5f99943f757d6 (127.27.254.2:46621), aafaecccab5a4f9ba061a99c5454dbff (127.27.254.1:45559)
I20260808 15:54:52.570293 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.564586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:46621 (local address 127.27.254.3:57563)
0808 15:54:52.564793 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:52.564814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:52.564933 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:52.565505 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:52.565510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:52.565543 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:52.565760 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.565772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.567106 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.567114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.568414 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.568426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.569502 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.569510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.569731 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.569763 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:54:52.569865 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:54:52.569993 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":4}
I20260808 15:54:52.570348 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.564818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:57563 (local address 127.27.254.2:46621)
0808 15:54:52.565006 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.565014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.565037 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:54:52.565114 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.565118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.565173 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.565308 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:52.565994 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.566987 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.568574 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.569322 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.569526 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.569604 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.569916 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.570051 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:54:52.571252 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.565204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:42755 (local address 127.27.254.1:45559)
0808 15:54:52.565460 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.565465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.565782 (+ 317us) server_negotiation.cc:407] Connection header received
0808 15:54:52.565945 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.565948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.566024 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.566093 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:52.566669 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.567345 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.569027 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.570024 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.570190 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.570261 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.570969 (+ 708us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.571081 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:54:52.571354 29721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.564967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45559 (local address 127.27.254.3:42755)
0808 15:54:52.565663 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:52.565685 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:52.565813 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:52.566131 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:52.566135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:52.566159 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:52.566528 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.566539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.567518 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.567522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.568857 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:52.568868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.570552 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.570559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:52.570776 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.570808 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:54:52.570909 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:54:52.571044 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260808 15:54:52.571415 29516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1519f8a41ca34bedad6782a45e8d1224" candidate_uuid: "6558e9afc70140278c9c1eee4a53cfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f18532e4c804a90b5b5f99943f757d6" is_pre_election: true
I20260808 15:54:52.572057 29516 raft_consensus.cc:2454] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6558e9afc70140278c9c1eee4a53cfa9 in term 0.
I20260808 15:54:52.572361 29382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1519f8a41ca34bedad6782a45e8d1224" candidate_uuid: "6558e9afc70140278c9c1eee4a53cfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aafaecccab5a4f9ba061a99c5454dbff" is_pre_election: true
I20260808 15:54:52.572858 29583 leader_election.cc:302] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f18532e4c804a90b5b5f99943f757d6, 6558e9afc70140278c9c1eee4a53cfa9; no voters:
I20260808 15:54:52.572907 29382 raft_consensus.cc:2454] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6558e9afc70140278c9c1eee4a53cfa9 in term 0.
I20260808 15:54:52.573261 29719 raft_consensus.cc:2781] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:54:52.573468 29719 raft_consensus.cc:482] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:54:52.573593 29719 raft_consensus.cc:3037] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:52.575836 29719 raft_consensus.cc:504] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.576438 29719 leader_election.cc:288] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [CANDIDATE]: Term 1 election: Requested vote from peers 3f18532e4c804a90b5b5f99943f757d6 (127.27.254.2:46621), aafaecccab5a4f9ba061a99c5454dbff (127.27.254.1:45559)
I20260808 15:54:52.576887 29516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1519f8a41ca34bedad6782a45e8d1224" candidate_uuid: "6558e9afc70140278c9c1eee4a53cfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f18532e4c804a90b5b5f99943f757d6"
I20260808 15:54:52.576946 29382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1519f8a41ca34bedad6782a45e8d1224" candidate_uuid: "6558e9afc70140278c9c1eee4a53cfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aafaecccab5a4f9ba061a99c5454dbff"
I20260808 15:54:52.577090 29516 raft_consensus.cc:3037] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:52.577176 29382 raft_consensus.cc:3037] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:52.579388 29516 raft_consensus.cc:2454] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6558e9afc70140278c9c1eee4a53cfa9 in term 1.
I20260808 15:54:52.579589 29382 raft_consensus.cc:2454] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6558e9afc70140278c9c1eee4a53cfa9 in term 1.
I20260808 15:54:52.579795 29583 leader_election.cc:302] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f18532e4c804a90b5b5f99943f757d6, 6558e9afc70140278c9c1eee4a53cfa9; no voters:
I20260808 15:54:52.580113 29719 raft_consensus.cc:2781] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:52.580559 29719 raft_consensus.cc:686] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [term 1 LEADER]: Becoming Leader. State: Replica: 6558e9afc70140278c9c1eee4a53cfa9, State: Running, Role: LEADER
I20260808 15:54:52.581131 29719 consensus_queue.cc:237] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } }
I20260808 15:54:52.586062 29237 catalog_manager.cc:5495] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6558e9afc70140278c9c1eee4a53cfa9 (127.27.254.3). New cstate: current_term: 1 leader_uuid: "6558e9afc70140278c9c1eee4a53cfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f18532e4c804a90b5b5f99943f757d6" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6558e9afc70140278c9c1eee4a53cfa9" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 41477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aafaecccab5a4f9ba061a99c5454dbff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45559 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:54:52.610621 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.596888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.27.254.1:45559)
0808 15:54:52.597041 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.597045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.597585 (+ 540us) server_negotiation.cc:407] Connection header received
0808 15:54:52.597728 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.597732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.597799 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.597880 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.599322 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.600029 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.601253 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.601712 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.601815 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.605607 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.605620 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.605636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.605669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.608259 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.608838 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.608845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.608847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.608960 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.609350 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.609353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.609355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.609739 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.609957 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.610216 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.610368 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 15:54:52.633694 29428 tablet.cc:2369] T 1519f8a41ca34bedad6782a45e8d1224 P aafaecccab5a4f9ba061a99c5454dbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260808 15:54:52.637451 29695 tablet.cc:2369] T 1519f8a41ca34bedad6782a45e8d1224 P 6558e9afc70140278c9c1eee4a53cfa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:54:52.640678 29562 tablet.cc:2369] T 1519f8a41ca34bedad6782a45e8d1224 P 3f18532e4c804a90b5b5f99943f757d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:54:52.656397 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.642751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.27.254.1:45559)
0808 15:54:52.642951 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.642960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.643356 (+ 396us) server_negotiation.cc:407] Connection header received
0808 15:54:52.643573 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.643577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.643641 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.643750 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.645084 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.645898 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.647172 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.647569 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.647707 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.651420 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.651425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.651428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.651460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.653903 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.654508 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.654518 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.654521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.654640 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.655033 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.655039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.655042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.655306 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.655566 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.655845 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.656063 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:54:52.700449 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.687286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.27.254.1:45559)
0808 15:54:52.687519 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.687526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.687705 (+ 179us) server_negotiation.cc:407] Connection header received
0808 15:54:52.687874 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.687878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.687946 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.688027 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.689386 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.690149 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.691351 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.691787 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.691886 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.695805 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.695811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.695814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.695848 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.698282 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.698867 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.698870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.698872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.698981 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.699291 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.699294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.699296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.699578 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.699761 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.699976 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.700161 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 15:54:52.702380 29342 tablet_service.cc:2908] Rejecting scan request for tablet 1519f8a41ca34bedad6782a45e8d1224: Uninitialized: clean time has not yet been initialized
I20260808 15:54:52.705286 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29299
I20260808 15:54:52.717733 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.706290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.27.254.1:45559)
0808 15:54:52.706498 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.706507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.706756 (+ 249us) server_negotiation.cc:407] Connection header received
0808 15:54:52.706884 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.706888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.706957 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.707033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.708114 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.708757 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.710052 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.710479 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.710589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.712968 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.712977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.712986 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.713030 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.715575 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.716143 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.716147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.716149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.716257 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.716531 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.716534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.716536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.716789 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.716989 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.717184 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.717333 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:54:52.718421 29422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:52.811018 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29299
I20260808 15:54:52.828420 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29433
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:54:52.842783 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.829255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.27.254.2:46621)
0808 15:54:52.829661 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.829666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.829683 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:54:52.829736 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.829739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.829801 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.829890 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.831168 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.831991 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.833328 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.833760 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.833852 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.837707 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.837720 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.837735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.837766 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.840334 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.841024 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.841031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.841033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.841148 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.841601 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.841604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.841606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.841959 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.842208 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.842378 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.842524 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 15:54:52.843413 29556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:52.929270 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29433
I20260808 15:54:52.946704 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29566
Aug 08 15:54:52 dist-test-slave-srgv krb5kdc[29193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:54:52.960862 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:52.947491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.27.254.3:41477)
0808 15:54:52.947666 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:52.947670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:52.947687 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:54:52.947743 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:52.947747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:52.947804 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:52.947885 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:52.949172 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.950000 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.951266 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:52.951641 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:52.951729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:52.955694 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:52.955709 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:52.955724 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:52.955758 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:52.958402 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.959130 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.959140 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.959143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.959271 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:52.959646 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:52.959649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:52.959651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:52.959985 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:52.960199 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:52.960426 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 15:54:52.960596 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:54:52.961608 29689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:53.049258 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29566
I20260808 15:54:53.066856 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29206
I20260808 15:54:53.068081 29268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:53.175971 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29206
2026-08-08T15:54:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4072 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:53 dist-test-slave-srgv krb5kdc[29741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:53 dist-test-slave-srgv krb5kdc[29741](info): set up 2 sockets
Aug 08 15:54:53 dist-test-slave-srgv krb5kdc[29741](info): commencing operation
krb5kdc: starting...
W20260808 15:54:54.259485 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:54:54 dist-test-slave-srgv krb5kdc[29741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:54:54.436524 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:42975
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:42975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:54.754163 29754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:54.754447 29754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:54.754623 29754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:54.761876 29754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:54.761977 29754 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:54:54.762072 29754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:54.762171 29754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:54.762238 29754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:54.762327 29754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:54.772306 29754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:42975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:42975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:54.773633 29754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:54.774909 29754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:54.786391 29759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:54.786329 29762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:54.786357 29760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:54.787135 29754 server_base.cc:1034] running on GCE node
I20260808 15:54:54.787942 29754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:54.789352 29754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:54.790609 29754 hybrid_clock.cc:648] HybridClock initialized: now 1786204494790543 us; error 40 us; skew 500 ppm
I20260808 15:54:54.790934 29754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:54 dist-test-slave-srgv krb5kdc[29741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204494, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:54.793906 29754 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:54:54.795293 29754 webserver.cc:460] Webserver started at http://127.27.254.62:45893/ using document root <none> and password file <none>
I20260808 15:54:54.795982 29754 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:54.796110 29754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:54.796428 29754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:54.799357 29754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf"
format_stamp: "Formatted at 2026-08-08 15:54:54 on dist-test-slave-srgv"
server_key: "2eca26f3f4add2434073bff815583502"
server_key_iv: "e3afe9d49667ba52e51ecd9bc03380ac"
server_key_version: "encryptionkey@0"
I20260808 15:54:54.800114 29754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf"
format_stamp: "Formatted at 2026-08-08 15:54:54 on dist-test-slave-srgv"
server_key: "2eca26f3f4add2434073bff815583502"
server_key_iv: "e3afe9d49667ba52e51ecd9bc03380ac"
server_key_version: "encryptionkey@0"
I20260808 15:54:54.805150 29754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:54.808535 29769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:54.809851 29754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 15:54:54.810026 29754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf"
format_stamp: "Formatted at 2026-08-08 15:54:54 on dist-test-slave-srgv"
server_key: "2eca26f3f4add2434073bff815583502"
server_key_iv: "e3afe9d49667ba52e51ecd9bc03380ac"
server_key_version: "encryptionkey@0"
I20260808 15:54:54.810256 29754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:54.829002 29754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:54.832942 29754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:54.833250 29754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:54.854592 29754 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:42975
I20260808 15:54:54.854600 29830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:42975 every 8 connection(s)
I20260808 15:54:54.855954 29754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:54.859730 29831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:54.863337 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29754
I20260808 15:54:54.863588 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:54:54.864424 28664 external_mini_cluster.cc:1428] Setting key 04e00cd9de87f8696a5995d23f721f28
I20260808 15:54:54.873191 29831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: Bootstrap starting.
I20260808 15:54:54.876888 29831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:54.878245 29831 log.cc:824] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:54:54 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:54.881189 29831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: No bootstrap required, opened a new log
I20260808 15:54:54.887486 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:54.866597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.27.254.62:42975)
0808 15:54:54.867372 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:54.867383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:54.867423 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:54:54.869068 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:54.869088 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:54.869570 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:54.869899 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:54.871606 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:54.872839 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:54.874544 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:54.875221 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:54.875354 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:54.880383 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:54.880390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:54.880404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:54.880439 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:54.883490 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:54.884285 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:54.884292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:54.884298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:54.884454 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:54.884826 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:54.884829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:54.884831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:54.885330 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:54.885639 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:54.886044 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 15:54:54.886430 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":365,"threads_started":1}
I20260808 15:54:54.890219 29831 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42975 } }
I20260808 15:54:54.890655 29831 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:54.890764 29831 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f15e68ca4a4ffea186e3d75bb0c4cf, State: Initialized, Role: FOLLOWER
I20260808 15:54:54.891341 29831 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42975 } }
I20260808 15:54:54.891597 29831 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:54.891750 29831 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:54.891920 29831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:54.894344 29831 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42975 } }
I20260808 15:54:54.894855 29831 leader_election.cc:302] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f15e68ca4a4ffea186e3d75bb0c4cf; no voters:
I20260808 15:54:54.895411 29831 leader_election.cc:288] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:54.895570 29836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:54.896616 29836 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [term 1 LEADER]: Becoming Leader. State: Replica: b1f15e68ca4a4ffea186e3d75bb0c4cf, State: Running, Role: LEADER
I20260808 15:54:54.897189 29836 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42975 } }
I20260808 15:54:54.897559 29831 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:54.900027 29838 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f15e68ca4a4ffea186e3d75bb0c4cf. Latest consensus state: current_term: 1 leader_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42975 } } }
I20260808 15:54:54.900573 29838 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:54.900117 29837 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f15e68ca4a4ffea186e3d75bb0c4cf" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42975 } } }
I20260808 15:54:54.900774 29837 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:54.902958 29845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:54.909137 29845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:54.917588 29845 catalog_manager.cc:1347] Generated new cluster ID: 2ca567629d8e4a0999421aac4c295413
I20260808 15:54:54.917703 29845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:54.929311 29845 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:54.930589 29845 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:54.937726 29845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: Generated new TSK 0
I20260808 15:54:54.938741 29845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:54:54.989224 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:42975
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:55.320515 29857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:55.320808 29857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:55.320971 29857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:55.328230 29857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:55.328336 29857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:55.328485 29857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:54:55.328609 29857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:54:55.338292 29857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:42975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:55.339563 29857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:55.340869 29857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:55.353755 29862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:55.353868 29865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:55.353811 29863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:55.354601 29857 server_base.cc:1034] running on GCE node
I20260808 15:54:55.355211 29857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:55.356102 29857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:55.357376 29857 hybrid_clock.cc:648] HybridClock initialized: now 1786204495357307 us; error 47 us; skew 500 ppm
I20260808 15:54:55.357731 29857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:55 dist-test-slave-srgv krb5kdc[29741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204495, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:55.360759 29857 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:54:55.362136 29857 webserver.cc:460] Webserver started at http://127.27.254.1:43585/ using document root <none> and password file <none>
I20260808 15:54:55.362771 29857 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:55.362852 29857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:55.363103 29857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:55.365839 29857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "fe0a085108af47299a03ef7177ef412c"
format_stamp: "Formatted at 2026-08-08 15:54:55 on dist-test-slave-srgv"
server_key: "886808c07389e60a6bb613566a1e3a37"
server_key_iv: "183aaaf62d6e14746b48bfc619f39fd9"
server_key_version: "encryptionkey@0"
I20260808 15:54:55.366604 29857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "fe0a085108af47299a03ef7177ef412c"
format_stamp: "Formatted at 2026-08-08 15:54:55 on dist-test-slave-srgv"
server_key: "886808c07389e60a6bb613566a1e3a37"
server_key_iv: "183aaaf62d6e14746b48bfc619f39fd9"
server_key_version: "encryptionkey@0"
I20260808 15:54:55.371804 29857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:54:55.375145 29872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:55.376447 29857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:54:55.376626 29857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "fe0a085108af47299a03ef7177ef412c"
format_stamp: "Formatted at 2026-08-08 15:54:55 on dist-test-slave-srgv"
server_key: "886808c07389e60a6bb613566a1e3a37"
server_key_iv: "183aaaf62d6e14746b48bfc619f39fd9"
server_key_version: "encryptionkey@0"
I20260808 15:54:55.376834 29857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:55.406066 29857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:55.410679 29857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:55.410957 29857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:55.413661 29857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:55.413784 29857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:55.413947 29857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:55.414047 29857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:54:55 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204495, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:55.440182 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.415360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.27.254.1:50741)
0808 15:54:55.416247 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:55.416295 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:55.418861 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:55.419763 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:55.419777 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:55.420775 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:55.421880 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.421914 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.422990 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.422999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.425739 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.425758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.426730 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.426741 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.427114 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.427998 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:55.428036 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:55.431362 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:55.434664 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.434675 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.434704 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.435525 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.436055 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.436061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.436071 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.436307 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.437038 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:55.437045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:55.437391 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 15:54:55.437806 (+ 415us) client_negotiation.cc:240] Negotiation successful
0808 15:54:55.438478 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":398,"threads_started":1}
I20260808 15:54:55.441354 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.414622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:50741 (local address 127.27.254.62:42975)
0808 15:54:55.415068 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:55.415075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:55.416434 (+ 1359us) server_negotiation.cc:407] Connection header received
0808 15:54:55.419166 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:55.419172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:55.419267 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:55.419390 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:55.422106 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.422817 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.425937 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.426447 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.426599 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.431595 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:55.431600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:55.431603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:55.431635 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:55.434509 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.435721 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.435727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.435731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.435915 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.436442 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.436447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.436450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.436791 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:55.436960 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:55.440936 (+ 3976us) server_negotiation.cc:299] Negotiation successful
0808 15:54:55.441109 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260808 15:54:55.463754 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.447930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:54937 (local address 127.27.254.62:42975)
0808 15:54:55.448087 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:55.448091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:55.448780 (+ 689us) server_negotiation.cc:407] Connection header received
0808 15:54:55.448987 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:55.448991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:55.449055 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:55.449157 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:55.450560 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.451783 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.454053 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.454868 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.455060 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.457747 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:55.457752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:55.457756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:55.457787 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:55.461026 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.461815 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.461820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.461824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.461949 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.462337 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.462342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.462345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.462613 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:55.462775 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:55.463187 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 15:54:55.463400 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:54:55.463909 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.448041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.27.254.1:54937)
0808 15:54:55.448684 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:55.448706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:55.448846 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:55.449182 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:55.449187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:55.449899 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:55.450350 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.450370 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.451991 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.452003 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.453848 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.453868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.455393 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.455406 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.455628 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.456465 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:55.456487 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:55.457494 (+ 1007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:55.461174 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.461180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.461184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.461683 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.462061 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.462066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.462070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.462202 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.462737 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:55.462748 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:55.462933 (+ 185us) client_negotiation.cc:769] Sending connection context
0808 15:54:55.463068 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 15:54:55.463395 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":344,"threads_started":1}
I20260808 15:54:55.471272 29857 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:44345
I20260808 15:54:55.471295 29992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:44345 every 8 connection(s)
I20260808 15:54:55.472520 29857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:54:55.474445 29993 heartbeater.cc:347] Connected to a master server at 127.27.254.62:42975
I20260808 15:54:55.474658 29993 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:55.475234 29993 heartbeater.cc:511] Master 127.27.254.62:42975 requested a full tablet report, sending...
I20260808 15:54:55.477035 29786 ts_manager.cc:194] Registered new tserver with Master: fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345)
I20260808 15:54:55.478551 29786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:50741
I20260808 15:54:55.479514 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29857
I20260808 15:54:55.479751 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:54:55.480511 28664 external_mini_cluster.cc:1428] Setting key a24222ea59a3cc20419c397c4034101d
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:54:55.517338 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:42975
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:55.833835 29997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:55.834095 29997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:55.834256 29997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:55.841295 29997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:55.841439 29997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:55.841583 29997 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:54:55.841733 29997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:54:55.851326 29997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:42975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:55.852566 29997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:55.853993 29997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:55.867017 30005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:55.866994 30003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:55.867517 30002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:55.867745 29997 server_base.cc:1034] running on GCE node
I20260808 15:54:55.868468 29997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:55.869534 29997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:55.870786 29997 hybrid_clock.cc:648] HybridClock initialized: now 1786204495870706 us; error 57 us; skew 500 ppm
I20260808 15:54:55.871096 29997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:55 dist-test-slave-srgv krb5kdc[29741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204495, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:55.874316 29997 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:54:55.875744 29997 webserver.cc:460] Webserver started at http://127.27.254.2:39111/ using document root <none> and password file <none>
I20260808 15:54:55.876405 29997 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:55.876488 29997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:55.876758 29997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:55.879580 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "1957e14960af42a29ade2da06193ac40"
format_stamp: "Formatted at 2026-08-08 15:54:55 on dist-test-slave-srgv"
server_key: "98728d737cfdac60edc59228d64d49d3"
server_key_iv: "63a231541b211d97346404ce8605a0b1"
server_key_version: "encryptionkey@0"
I20260808 15:54:55.880350 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "1957e14960af42a29ade2da06193ac40"
format_stamp: "Formatted at 2026-08-08 15:54:55 on dist-test-slave-srgv"
server_key: "98728d737cfdac60edc59228d64d49d3"
server_key_iv: "63a231541b211d97346404ce8605a0b1"
server_key_version: "encryptionkey@0"
I20260808 15:54:55.885344 29997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:54:55.888649 30012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:55.890113 29997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:55.890296 29997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "1957e14960af42a29ade2da06193ac40"
format_stamp: "Formatted at 2026-08-08 15:54:55 on dist-test-slave-srgv"
server_key: "98728d737cfdac60edc59228d64d49d3"
server_key_iv: "63a231541b211d97346404ce8605a0b1"
server_key_version: "encryptionkey@0"
I20260808 15:54:55.890512 29997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:55.906198 29997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:55.910115 29997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:55.910408 29997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:55.913009 29997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:55.913120 29997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:55.913306 29997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:55.913388 29997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:54:55 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204495, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:55.939592 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.913742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44459 (local address 127.27.254.62:42975)
0808 15:54:55.913945 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:55.913951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:55.915495 (+ 1544us) server_negotiation.cc:407] Connection header received
0808 15:54:55.918536 (+ 3041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:55.918543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:55.918681 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:55.918808 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:55.922067 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.923120 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.925370 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.926193 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.926351 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.931143 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:55.931152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:55.931170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:55.931226 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:55.935537 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.936564 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.936573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.936579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.936759 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.937268 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.937273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.937277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.937708 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:55.937847 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:55.939055 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0808 15:54:55.939275 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:54:55.940991 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.914356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.27.254.2:44459)
0808 15:54:55.915306 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:55.915360 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:55.918174 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:55.919058 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:55.919066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:55.919819 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:55.920724 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.920749 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.923302 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.923309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.925196 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.925215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.926456 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.926466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.926747 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.927617 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:55.927655 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:55.930862 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:55.935689 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.935703 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.935727 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.936420 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.936902 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.936911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.936915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.937132 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.937827 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:55.937835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:55.938333 (+ 498us) client_negotiation.cc:769] Sending connection context
0808 15:54:55.938655 (+ 322us) client_negotiation.cc:240] Negotiation successful
0808 15:54:55.939272 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":340,"threads_started":1}
I20260808 15:54:55.963342 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.948790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:55899 (local address 127.27.254.62:42975)
0808 15:54:55.949001 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:55.949010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:55.949146 (+ 136us) server_negotiation.cc:407] Connection header received
0808 15:54:55.949388 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:55.949396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:55.949491 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:55.949552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:55.950908 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.951960 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.953781 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.954426 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.954511 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.956773 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:55.956780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:55.956785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:55.956820 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:55.960100 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.960927 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.960934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.960940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.961089 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:55.961611 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:55.961616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:55.961619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:55.962008 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:55.962119 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:55.962700 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 15:54:55.962952 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:54:55.964383 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:55.948385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.27.254.2:55899)
0808 15:54:55.949036 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:55.949055 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:55.949243 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:55.949757 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:55.949764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:55.950389 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:55.950740 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.950752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.952142 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.952147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.953621 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:55.953635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:55.954798 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:55.954803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:55.954966 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:55.955624 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:55.955645 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:55.956558 (+ 913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:55.960274 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.960279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.960283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.960783 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.961280 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:55.961291 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:55.961294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:55.961482 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:55.962247 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:55.962251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:55.962434 (+ 183us) client_negotiation.cc:769] Sending connection context
0808 15:54:55.963615 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0808 15:54:55.963960 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":618,"threads_started":1}
I20260808 15:54:55.974102 29997 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:39661
I20260808 15:54:55.974188 30131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:39661 every 8 connection(s)
I20260808 15:54:55.975383 29997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:54:55.977324 30132 heartbeater.cc:347] Connected to a master server at 127.27.254.62:42975
I20260808 15:54:55.977568 30132 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:55.978196 30132 heartbeater.cc:511] Master 127.27.254.62:42975 requested a full tablet report, sending...
I20260808 15:54:55.979576 29786 ts_manager.cc:194] Registered new tserver with Master: 1957e14960af42a29ade2da06193ac40 (127.27.254.2:39661)
I20260808 15:54:55.980427 29786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:44459
I20260808 15:54:55.985553 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 29997
I20260808 15:54:55.985733 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:54:55.986114 28664 external_mini_cluster.cc:1428] Setting key b258a75956d7864ac7efb802fc6763f9
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:54:56.024926 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:42975
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:56.346767 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:56.347046 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:56.347215 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:56.354435 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:56.354539 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:56.354713 30136 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:54:56.354828 30136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:54:56.364486 30136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:42975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:56.365767 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:56.367120 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:56.380133 30141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:56.380195 30144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:56.380138 30142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:56.380935 30136 server_base.cc:1034] running on GCE node
I20260808 15:54:56.381721 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:56.382558 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:56.383774 30136 hybrid_clock.cc:648] HybridClock initialized: now 1786204496383703 us; error 47 us; skew 500 ppm
I20260808 15:54:56.384078 30136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:56 dist-test-slave-srgv krb5kdc[29741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:56.387194 30136 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:54:56.388710 30136 webserver.cc:460] Webserver started at http://127.27.254.3:38027/ using document root <none> and password file <none>
I20260808 15:54:56.389477 30136 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:56.389611 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:56.389967 30136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:56.392972 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "276d84f70d014349ab9f78a11a6360ae"
format_stamp: "Formatted at 2026-08-08 15:54:56 on dist-test-slave-srgv"
server_key: "b5e0898ad012044b4d470e55885e51ab"
server_key_iv: "dba72287cba66d5eb2cc375b6ba5fa1f"
server_key_version: "encryptionkey@0"
I20260808 15:54:56.393862 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "276d84f70d014349ab9f78a11a6360ae"
format_stamp: "Formatted at 2026-08-08 15:54:56 on dist-test-slave-srgv"
server_key: "b5e0898ad012044b4d470e55885e51ab"
server_key_iv: "dba72287cba66d5eb2cc375b6ba5fa1f"
server_key_version: "encryptionkey@0"
I20260808 15:54:56.399291 30136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260808 15:54:56.402957 30151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:56.404253 30136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:54:56.404426 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "276d84f70d014349ab9f78a11a6360ae"
format_stamp: "Formatted at 2026-08-08 15:54:56 on dist-test-slave-srgv"
server_key: "b5e0898ad012044b4d470e55885e51ab"
server_key_iv: "dba72287cba66d5eb2cc375b6ba5fa1f"
server_key_version: "encryptionkey@0"
I20260808 15:54:56.404642 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:56.424919 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:56.428920 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:56.429196 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:56.432020 30136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:56.432133 30136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:56.432301 30136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:56.432379 30136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:54:56 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:56.455895 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.432446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:58111 (local address 127.27.254.62:42975)
0808 15:54:56.432651 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.432658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.433950 (+ 1292us) server_negotiation.cc:407] Connection header received
0808 15:54:56.435945 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.435951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.436042 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.436157 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:56.438564 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.439678 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.442264 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.443053 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.443204 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.448295 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:56.448303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:56.448313 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:56.448348 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:56.451880 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:56.452979 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:56.452987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:56.452991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:56.453159 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:56.453797 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:56.453802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:56.453806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:56.454183 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:56.454357 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.455464 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.455642 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:54:56.458055 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.432883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.27.254.3:58111)
0808 15:54:56.433770 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.433819 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.435643 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.436399 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.436407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.437227 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:56.438273 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.438308 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.439841 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.439848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.442078 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.442095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.443265 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.443269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.443623 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.444454 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:56.444502 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:56.448039 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:56.452061 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:56.452071 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:56.452084 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:56.452793 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:56.453259 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:56.453264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:56.453268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:56.453626 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:56.454363 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:56.454376 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:56.454969 (+ 593us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.455376 (+ 407us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.456099 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":368,"threads_started":1}
I20260808 15:54:56.481726 29993 heartbeater.cc:503] Master 127.27.254.62:42975 was elected leader, sending a full tablet report...
I20260808 15:54:56.482949 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.465587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.27.254.3:50563)
0808 15:54:56.466249 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.466274 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.466451 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.466890 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.466897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.467712 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:56.468289 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.468311 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.469764 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.469768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.471411 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.471426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.472802 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.472809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.472988 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.473779 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:56.473802 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:56.474851 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:56.479588 (+ 4737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:56.479596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:56.479605 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:56.480302 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:56.480798 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:56.480808 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:56.480812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:56.480995 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:56.481759 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:56.481766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:56.481965 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.482135 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.482497 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":374,"threads_started":1}
I20260808 15:54:56.483163 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.465568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:50563 (local address 127.27.254.62:42975)
0808 15:54:56.465807 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.465816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.466374 (+ 558us) server_negotiation.cc:407] Connection header received
0808 15:54:56.466607 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.466616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.466719 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.466866 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:56.468483 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.469602 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.471610 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.472534 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.472745 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.475082 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:56.475092 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:56.475098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:56.475151 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:56.479425 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:56.480468 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:56.480473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:56.480477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:56.480645 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:56.481154 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:56.481159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:56.481162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:56.481614 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:56.481772 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.482286 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.482596 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:54:56.486719 30136 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:45757
I20260808 15:54:56.486727 30270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:45757 every 8 connection(s)
I20260808 15:54:56.488101 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:54:56.490036 30271 heartbeater.cc:347] Connected to a master server at 127.27.254.62:42975
I20260808 15:54:56.490238 30271 heartbeater.cc:464] Registering TS with master...
I20260808 15:54:56.490799 30271 heartbeater.cc:511] Master 127.27.254.62:42975 requested a full tablet report, sending...
I20260808 15:54:56.492017 29786 ts_manager.cc:194] Registered new tserver with Master: 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757)
I20260808 15:54:56.492242 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 30136
I20260808 15:54:56.492445 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:54:56.492870 28664 external_mini_cluster.cc:1428] Setting key 9fcaa3a0fa382e61676d247fa2747b81
I20260808 15:54:56.492949 29786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:58111
I20260808 15:54:56.496011 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:54:56 dist-test-slave-srgv krb5kdc[29741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:54:56 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:56.541198 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.527435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.27.254.62:42975)
0808 15:54:56.527694 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.527701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.527965 (+ 264us) server_negotiation.cc:407] Connection header received
0808 15:54:56.528201 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.528207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.528281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.528389 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:56.529901 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.530645 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.531934 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.532354 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.532490 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.536410 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:56.536415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:56.536419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:56.536452 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:56.538928 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:56.539508 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:56.539512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:56.539514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:56.539632 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:56.539971 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:56.539975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:56.539977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:56.540232 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:56.540415 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.540630 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.540803 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:54:56.545981 29786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:54:56.552613 29786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:54:56.577250 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.564867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:39661 (local address 127.0.0.1:60268)
0808 15:54:56.568358 (+ 3491us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.568381 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.568522 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.569349 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.569359 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.569396 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:56.569830 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.569843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.571679 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.571684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.574240 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.574255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.576082 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.576088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.576318 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.576378 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.576488 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.576673 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3310,"mutex_wait_us":49,"thread_start_us":3128,"threads_started":1}
I20260808 15:54:56.577822 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.564848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:45757 (local address 127.0.0.1:36296)
0808 15:54:56.567484 (+ 2636us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.567527 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.567714 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.570453 (+ 2739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.570458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.570486 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:56.570985 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.570998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.573206 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.573212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.575314 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.575331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.576867 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.576874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.577066 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.577120 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.577221 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.577361 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2407,"mutex_wait_us":32,"thread_start_us":2171,"threads_started":1}
I20260808 15:54:56.577913 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.565490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.27.254.2:39661)
0808 15:54:56.566154 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.566166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.568658 (+ 2492us) server_negotiation.cc:407] Connection header received
0808 15:54:56.568781 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.568792 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.569069 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.569836 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:56.570013 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.571523 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.574433 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.575861 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.577141 (+ 1280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.577334 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.577415 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.577620 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":267,"threads_started":1}
I20260808 15:54:56.578375 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.565985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.27.254.1:44345)
0808 15:54:56.568633 (+ 2648us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.568642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.569697 (+ 1055us) server_negotiation.cc:407] Connection header received
0808 15:54:56.569800 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.569807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.570020 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.570237 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:56.570991 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.572569 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.575258 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.576691 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.576832 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.577031 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.577911 (+ 880us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.578083 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2466,"thread_start_us":2324,"threads_started":1}
I20260808 15:54:56.578547 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.565560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.27.254.3:45757)
0808 15:54:56.569833 (+ 4273us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.569841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.569873 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:54:56.569971 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.569981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.570218 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.570455 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:56.571221 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.573060 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.575492 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.576649 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.576794 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.577044 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.578063 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.578246 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4104,"thread_start_us":3880,"threads_started":1}
I20260808 15:54:56.578845 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.566394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44345 (local address 127.0.0.1:35170)
0808 15:54:56.569510 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.569529 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.569663 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.570237 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.570245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.570273 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:56.570827 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.570838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.572722 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.572729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.575072 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.575087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.577211 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.577220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.577432 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.577464 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.577548 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.577691 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2918,"mutex_wait_us":9,"thread_start_us":2420,"threads_started":1}
I20260808 15:54:56.581887 30201 tablet_service.cc:1467] Processing CreateTablet for tablet b6351a681d61417bb3d5789521db92b3 (DEFAULT_TABLE table=test-table [id=1e2f307dea064812b564d4741294ce3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:56.582006 29923 tablet_service.cc:1467] Processing CreateTablet for tablet b6351a681d61417bb3d5789521db92b3 (DEFAULT_TABLE table=test-table [id=1e2f307dea064812b564d4741294ce3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:56.581933 30062 tablet_service.cc:1467] Processing CreateTablet for tablet b6351a681d61417bb3d5789521db92b3 (DEFAULT_TABLE table=test-table [id=1e2f307dea064812b564d4741294ce3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:56.583137 30201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6351a681d61417bb3d5789521db92b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:56.583252 29923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6351a681d61417bb3d5789521db92b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:56.583257 30062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6351a681d61417bb3d5789521db92b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:56.594672 30290 tablet_bootstrap.cc:492] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Bootstrap starting.
I20260808 15:54:56.597693 30292 tablet_bootstrap.cc:492] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Bootstrap starting.
I20260808 15:54:56.599694 30290 tablet_bootstrap.cc:654] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:56.601055 30290 log.cc:824] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:56.601266 30291 tablet_bootstrap.cc:492] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Bootstrap starting.
I20260808 15:54:56.602562 30292 tablet_bootstrap.cc:654] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:56.603825 30292 log.cc:824] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:56.603895 30290 tablet_bootstrap.cc:492] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: No bootstrap required, opened a new log
I20260808 15:54:56.604167 30290 ts_tablet_manager.cc:1397] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 15:54:56.604815 30291 tablet_bootstrap.cc:654] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:56.606045 30291 log.cc:824] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:56.607107 30292 tablet_bootstrap.cc:492] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: No bootstrap required, opened a new log
I20260808 15:54:56.607800 30292 ts_tablet_manager.cc:1397] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260808 15:54:56.609465 30291 tablet_bootstrap.cc:492] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: No bootstrap required, opened a new log
I20260808 15:54:56.609732 30291 ts_tablet_manager.cc:1397] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:54:56.612612 30290 raft_consensus.cc:348] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.612991 30290 raft_consensus.cc:374] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:56.613096 30290 raft_consensus.cc:729] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276d84f70d014349ab9f78a11a6360ae, State: Initialized, Role: FOLLOWER
I20260808 15:54:56.613762 30290 consensus_queue.cc:260] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.614905 30271 heartbeater.cc:503] Master 127.27.254.62:42975 was elected leader, sending a full tablet report...
I20260808 15:54:56.615608 30290 ts_tablet_manager.cc:1428] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 15:54:56.618692 30292 raft_consensus.cc:348] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.619130 30292 raft_consensus.cc:374] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:56.619239 30292 raft_consensus.cc:729] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a085108af47299a03ef7177ef412c, State: Initialized, Role: FOLLOWER
I20260808 15:54:56.619860 30292 consensus_queue.cc:260] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.620252 30291 raft_consensus.cc:348] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.620697 30291 raft_consensus.cc:374] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:56.620800 30291 raft_consensus.cc:729] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1957e14960af42a29ade2da06193ac40, State: Initialized, Role: FOLLOWER
I20260808 15:54:56.621363 30291 consensus_queue.cc:260] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.621816 30292 ts_tablet_manager.cc:1428] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260808 15:54:56.622588 30132 heartbeater.cc:503] Master 127.27.254.62:42975 was elected leader, sending a full tablet report...
I20260808 15:54:56.623168 30291 ts_tablet_manager.cc:1428] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260808 15:54:56.658759 30298 raft_consensus.cc:482] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:54:56.659106 30298 raft_consensus.cc:504] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.660931 30298 leader_election.cc:288] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757), fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345)
I20260808 15:54:56.668744 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.661561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:60423 (local address 127.27.254.1:44345)
0808 15:54:56.661804 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.661812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.662403 (+ 591us) server_negotiation.cc:407] Connection header received
0808 15:54:56.662754 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.662761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.662848 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.662918 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:56.663576 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.664290 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.666272 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.667461 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.667616 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.667711 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.668342 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.668484 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:54:56.668732 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.661256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:39531 (local address 127.27.254.3:45757)
0808 15:54:56.661509 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.661518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.662624 (+ 1106us) server_negotiation.cc:407] Connection header received
0808 15:54:56.662865 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.662869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.662959 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.663056 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:54:56.663680 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.664382 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.666193 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.667437 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.667621 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.667708 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.668342 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.668484 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:54:56.668974 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.661427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:45757 (local address 127.27.254.2:39531)
0808 15:54:56.662509 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.662526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.662693 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.663105 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.663108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.663139 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:56.663536 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.663547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.664524 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.664527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.666038 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.666052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.667754 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.667761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.668056 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.668193 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.668292 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.668484 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":166,"thread_start_us":750,"threads_started":1}
I20260808 15:54:56.669090 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.661346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44345 (local address 127.27.254.2:60423)
0808 15:54:56.662242 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:56.662276 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:56.662550 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:56.662959 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:56.662964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:56.662989 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:54:56.663420 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.663430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.664417 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.664425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.666065 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:56.666078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.667758 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.667763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:56.668059 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.668192 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 15:54:56.668292 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:54:56.668484 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":148,"thread_start_us":508,"threads_started":1}
I20260808 15:54:56.670053 30221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6351a681d61417bb3d5789521db92b3" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276d84f70d014349ab9f78a11a6360ae" is_pre_election: true
I20260808 15:54:56.670172 29944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6351a681d61417bb3d5789521db92b3" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a085108af47299a03ef7177ef412c" is_pre_election: true
I20260808 15:54:56.670612 30221 raft_consensus.cc:2454] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 0.
I20260808 15:54:56.670643 29944 raft_consensus.cc:2454] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 0.
I20260808 15:54:56.671239 30013 leader_election.cc:302] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1957e14960af42a29ade2da06193ac40, fe0a085108af47299a03ef7177ef412c; no voters:
I20260808 15:54:56.671659 30298 raft_consensus.cc:2781] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:54:56.671821 30298 raft_consensus.cc:482] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:54:56.671931 30298 raft_consensus.cc:3037] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:56.674127 30298 raft_consensus.cc:504] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.674700 30298 leader_election.cc:288] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 election: Requested vote from peers 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757), fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345)
I20260808 15:54:56.675055 30221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6351a681d61417bb3d5789521db92b3" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276d84f70d014349ab9f78a11a6360ae"
I20260808 15:54:56.675093 29944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6351a681d61417bb3d5789521db92b3" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a085108af47299a03ef7177ef412c"
I20260808 15:54:56.675238 30221 raft_consensus.cc:3037] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:56.675305 29944 raft_consensus.cc:3037] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:56.677511 29944 raft_consensus.cc:2454] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 1.
I20260808 15:54:56.677541 30221 raft_consensus.cc:2454] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 1.
I20260808 15:54:56.677913 30014 leader_election.cc:302] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1957e14960af42a29ade2da06193ac40, 276d84f70d014349ab9f78a11a6360ae; no voters:
I20260808 15:54:56.678233 30298 raft_consensus.cc:2781] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:56.678696 30298 raft_consensus.cc:686] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 1 LEADER]: Becoming Leader. State: Replica: 1957e14960af42a29ade2da06193ac40, State: Running, Role: LEADER
I20260808 15:54:56.679132 30298 consensus_queue.cc:237] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } }
I20260808 15:54:56.683848 29786 catalog_manager.cc:5495] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1957e14960af42a29ade2da06193ac40 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } health_report { overall_health: UNKNOWN } } }
I20260808 15:54:56.709116 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:56.703960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.27.254.2:39661)
0808 15:54:56.704178 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:56.704183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:56.704201 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:54:56.704455 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:56.704458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:56.704527 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:56.704636 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:56.704939 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.705765 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.707034 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:56.707411 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:56.707498 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:56.707947 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:56.708507 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:56.708607 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:56.708737 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 15:54:56.708855 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:54:56.716820 30221 raft_consensus.cc:1261] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Refusing update from remote peer 1957e14960af42a29ade2da06193ac40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:56.716816 29944 raft_consensus.cc:1261] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Refusing update from remote peer 1957e14960af42a29ade2da06193ac40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:56.717803 30301 consensus_queue.cc:1035] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:56.718259 30298 consensus_queue.cc:1035] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:54:56.724965 29994 tablet.cc:2369] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:54:56.727236 30133 tablet.cc:2369] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:54:56.728468 30304 mvcc.cc:205] Tried to move back new op lower bound from 7316293618543267840 to 7316293618401783808. Current Snapshot: MvccSnapshot[applied={T|T < 7316293618543267840}]
I20260808 15:54:56.749490 29785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:54:56.749773 29785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:54:56.753763 29785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: Sending DeleteTablet for 3 replicas of tablet b6351a681d61417bb3d5789521db92b3
I20260808 15:54:56.755247 29923 tablet_service.cc:1514] Processing DeleteTablet for tablet b6351a681d61417bb3d5789521db92b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:54:56 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:35170
I20260808 15:54:56.755208 30062 tablet_service.cc:1514] Processing DeleteTablet for tablet b6351a681d61417bb3d5789521db92b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:54:56 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:60268
I20260808 15:54:56.757313 30201 tablet_service.cc:1514] Processing DeleteTablet for tablet b6351a681d61417bb3d5789521db92b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:54:56 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:36296
I20260808 15:54:56.757530 29784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:54:56.758590 29784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:54:56.766698 30314 tablet_replica.cc:330] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: stopping tablet replica
I20260808 15:54:56.767334 30314 raft_consensus.cc:2229] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:54:56.768071 30314 raft_consensus.cc:2258] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:54:56.768344 29922 tablet_service.cc:1467] Processing CreateTablet for tablet 6765292c71c14143b58cfcaf11a70932 (DEFAULT_TABLE table=test-table [id=f56e3cc4f3f14b47a565f246b7c7f6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:56.768436 30200 tablet_service.cc:1467] Processing CreateTablet for tablet 6765292c71c14143b58cfcaf11a70932 (DEFAULT_TABLE table=test-table [id=f56e3cc4f3f14b47a565f246b7c7f6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:56.768932 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6765292c71c14143b58cfcaf11a70932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:56.768932 29922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6765292c71c14143b58cfcaf11a70932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:56.770445 30314 ts_tablet_manager.cc:1905] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:54:56.774502 30062 tablet_service.cc:1467] Processing CreateTablet for tablet 6765292c71c14143b58cfcaf11a70932 (DEFAULT_TABLE table=test-table [id=f56e3cc4f3f14b47a565f246b7c7f6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:54:56.774966 30062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6765292c71c14143b58cfcaf11a70932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:56.775571 30290 tablet_bootstrap.cc:492] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Bootstrap starting.
I20260808 15:54:56.776160 30315 tablet_replica.cc:330] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: stopping tablet replica
I20260808 15:54:56.776644 30315 raft_consensus.cc:2229] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:54:56.777136 30315 raft_consensus.cc:2258] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:54:56.777262 30290 tablet_bootstrap.cc:654] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:56.778115 30314 ts_tablet_manager.cc:1918] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:54:56.778295 30314 log.cc:1196] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/b6351a681d61417bb3d5789521db92b3
I20260808 15:54:56.778921 30314 ts_tablet_manager.cc:1939] T b6351a681d61417bb3d5789521db92b3 P 1957e14960af42a29ade2da06193ac40: Deleting consensus metadata
I20260808 15:54:56.780447 30313 tablet_replica.cc:330] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: stopping tablet replica
I20260808 15:54:56.780946 30315 ts_tablet_manager.cc:1905] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:54:56.781020 30313 raft_consensus.cc:2229] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:54:56.781610 30313 raft_consensus.cc:2258] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:54:56.783651 30313 ts_tablet_manager.cc:1905] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:54:56.787177 30292 tablet_bootstrap.cc:492] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Bootstrap starting.
I20260808 15:54:56.788923 30292 tablet_bootstrap.cc:654] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:56.789783 30315 ts_tablet_manager.cc:1918] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:54:56.789954 30315 log.cc:1196] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/b6351a681d61417bb3d5789521db92b3
I20260808 15:54:56.790544 30315 ts_tablet_manager.cc:1939] T b6351a681d61417bb3d5789521db92b3 P 276d84f70d014349ab9f78a11a6360ae: Deleting consensus metadata
I20260808 15:54:56.791186 30313 ts_tablet_manager.cc:1918] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:54:56.791368 30313 log.cc:1196] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/b6351a681d61417bb3d5789521db92b3
I20260808 15:54:56.791992 30313 ts_tablet_manager.cc:1939] T b6351a681d61417bb3d5789521db92b3 P fe0a085108af47299a03ef7177ef412c: Deleting consensus metadata
I20260808 15:54:56.793392 30291 tablet_bootstrap.cc:492] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Bootstrap starting.
I20260808 15:54:56.795207 30291 tablet_bootstrap.cc:654] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:56.794400 29772 catalog_manager.cc:4841] TS 1957e14960af42a29ade2da06193ac40 (127.27.254.2:39661): tablet b6351a681d61417bb3d5789521db92b3 (table test-table [id=1e2f307dea064812b564d4741294ce3b]) successfully deleted
I20260808 15:54:56.795835 29770 catalog_manager.cc:4841] TS fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345): tablet b6351a681d61417bb3d5789521db92b3 (table test-table [id=1e2f307dea064812b564d4741294ce3b]) successfully deleted
I20260808 15:54:56.796065 30290 tablet_bootstrap.cc:492] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: No bootstrap required, opened a new log
I20260808 15:54:56.796224 30290 ts_tablet_manager.cc:1397] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.005s
I20260808 15:54:56.796838 29771 catalog_manager.cc:4841] TS 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757): tablet b6351a681d61417bb3d5789521db92b3 (table test-table [id=1e2f307dea064812b564d4741294ce3b]) successfully deleted
I20260808 15:54:56.797226 30292 tablet_bootstrap.cc:492] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: No bootstrap required, opened a new log
I20260808 15:54:56.797101 30290 raft_consensus.cc:348] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.797338 30290 raft_consensus.cc:374] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:56.797370 30292 ts_tablet_manager.cc:1397] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260808 15:54:56.797468 30290 raft_consensus.cc:729] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276d84f70d014349ab9f78a11a6360ae, State: Initialized, Role: FOLLOWER
I20260808 15:54:56.797813 30290 consensus_queue.cc:260] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.798928 30292 raft_consensus.cc:348] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.799147 30292 raft_consensus.cc:374] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:56.799250 30292 raft_consensus.cc:729] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a085108af47299a03ef7177ef412c, State: Initialized, Role: FOLLOWER
I20260808 15:54:56.799585 30291 tablet_bootstrap.cc:492] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: No bootstrap required, opened a new log
I20260808 15:54:56.799539 30292 consensus_queue.cc:260] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.799827 30291 ts_tablet_manager.cc:1397] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260808 15:54:56.800669 30291 raft_consensus.cc:348] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.800954 30291 raft_consensus.cc:374] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:56.800961 30292 ts_tablet_manager.cc:1428] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:54:56.801132 30291 raft_consensus.cc:729] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1957e14960af42a29ade2da06193ac40, State: Initialized, Role: FOLLOWER
I20260808 15:54:56.801177 30290 ts_tablet_manager.cc:1428] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260808 15:54:56.801508 30291 consensus_queue.cc:260] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.802892 30291 ts_tablet_manager.cc:1428] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:54:56.931005 30301 raft_consensus.cc:482] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:54:56.931316 30301 raft_consensus.cc:504] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.932096 30301 leader_election.cc:288] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345), 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757)
I20260808 15:54:56.932726 30220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6765292c71c14143b58cfcaf11a70932" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276d84f70d014349ab9f78a11a6360ae" is_pre_election: true
I20260808 15:54:56.932761 29944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6765292c71c14143b58cfcaf11a70932" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a085108af47299a03ef7177ef412c" is_pre_election: true
I20260808 15:54:56.933018 30220 raft_consensus.cc:2454] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 0.
I20260808 15:54:56.933037 29944 raft_consensus.cc:2454] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 0.
I20260808 15:54:56.933534 30013 leader_election.cc:302] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1957e14960af42a29ade2da06193ac40, fe0a085108af47299a03ef7177ef412c; no voters:
I20260808 15:54:56.933840 30301 raft_consensus.cc:2781] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:54:56.933955 30301 raft_consensus.cc:482] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:54:56.934110 30301 raft_consensus.cc:3037] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:56.936309 30301 raft_consensus.cc:504] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.936980 30301 leader_election.cc:288] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 election: Requested vote from peers fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345), 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757)
I20260808 15:54:56.937485 29944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6765292c71c14143b58cfcaf11a70932" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a085108af47299a03ef7177ef412c"
I20260808 15:54:56.937527 30220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6765292c71c14143b58cfcaf11a70932" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276d84f70d014349ab9f78a11a6360ae"
I20260808 15:54:56.937716 29944 raft_consensus.cc:3037] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:56.937736 30220 raft_consensus.cc:3037] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:56.940686 30220 raft_consensus.cc:2454] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 1.
I20260808 15:54:56.940690 29944 raft_consensus.cc:2454] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 1.
I20260808 15:54:56.941107 30013 leader_election.cc:302] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1957e14960af42a29ade2da06193ac40, fe0a085108af47299a03ef7177ef412c; no voters:
I20260808 15:54:56.941378 30301 raft_consensus.cc:2781] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:56.941577 30301 raft_consensus.cc:686] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 1 LEADER]: Becoming Leader. State: Replica: 1957e14960af42a29ade2da06193ac40, State: Running, Role: LEADER
I20260808 15:54:56.941836 30301 consensus_queue.cc:237] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:56.944674 29785 catalog_manager.cc:5495] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1957e14960af42a29ade2da06193ac40 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } health_report { overall_health: UNKNOWN } } }
W20260808 15:54:56.990382 30272 tablet.cc:2369] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204494, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:57.031350 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.012411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.27.254.62:42975)
0808 15:54:57.012640 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.012648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.013278 (+ 630us) server_negotiation.cc:407] Connection header received
0808 15:54:57.013550 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.013554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.013668 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.013770 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:57.015162 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.015894 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.017176 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.017682 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.017826 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.025989 (+ 8163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:57.025995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:57.025998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:57.026026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:57.028557 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.029242 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.029250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.029258 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.029378 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.029809 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.029817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.029822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.030117 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:57.030287 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.030695 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.031041 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":74}
I20260808 15:54:57.031388 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.012436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42975 (local address 127.0.0.1:48858)
0808 15:54:57.013145 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:57.013170 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:57.013361 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:57.013796 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:57.013800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:57.014681 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:57.015005 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:57.015015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.016023 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.016031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:57.017023 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:57.017035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.017868 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.017874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:57.018036 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.023315 (+ 5279us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:57.023333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:57.025802 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:57.028712 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:57.028725 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:57.028733 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:57.029118 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:57.029543 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:57.029549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:57.029552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:57.029705 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:57.030256 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:57.030265 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:57.030435 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 15:54:57.030607 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 15:54:57.030912 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":413,"threads_started":1}
I20260808 15:54:57.037113 29785 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:48858:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:54:57.038228 29785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:54:57.045208 29922 tablet_service.cc:1467] Processing CreateTablet for tablet 8e8d59cde4da443c8b6572022d4562c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=580d49e1a2d04361972e411ee4f44b16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:54:57.045629 30062 tablet_service.cc:1467] Processing CreateTablet for tablet 8e8d59cde4da443c8b6572022d4562c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=580d49e1a2d04361972e411ee4f44b16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:54:57.045843 29922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e8d59cde4da443c8b6572022d4562c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:57.045868 30201 tablet_service.cc:1467] Processing CreateTablet for tablet 8e8d59cde4da443c8b6572022d4562c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=580d49e1a2d04361972e411ee4f44b16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:54:57.046211 30062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e8d59cde4da443c8b6572022d4562c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:57.046381 30201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e8d59cde4da443c8b6572022d4562c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:57.054216 30290 tablet_bootstrap.cc:492] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: Bootstrap starting.
I20260808 15:54:57.054288 30292 tablet_bootstrap.cc:492] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: Bootstrap starting.
I20260808 15:54:57.055616 30290 tablet_bootstrap.cc:654] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:57.056211 30292 tablet_bootstrap.cc:654] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:57.057289 30291 tablet_bootstrap.cc:492] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: Bootstrap starting.
I20260808 15:54:57.059090 30291 tablet_bootstrap.cc:654] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:57.062054 30292 tablet_bootstrap.cc:492] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: No bootstrap required, opened a new log
I20260808 15:54:57.062317 30292 ts_tablet_manager.cc:1397] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260808 15:54:57.063149 30291 tablet_bootstrap.cc:492] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: No bootstrap required, opened a new log
I20260808 15:54:57.063175 30292 raft_consensus.cc:348] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.063570 30292 raft_consensus.cc:374] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:57.063719 30292 raft_consensus.cc:729] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a085108af47299a03ef7177ef412c, State: Initialized, Role: FOLLOWER
I20260808 15:54:57.063838 30291 ts_tablet_manager.cc:1397] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 15:54:57.064129 30292 consensus_queue.cc:260] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.064651 30291 raft_consensus.cc:348] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.064672 30297 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.065011 30297 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.064980 30291 raft_consensus.cc:374] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:57.065088 30291 raft_consensus.cc:729] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1957e14960af42a29ade2da06193ac40, State: Initialized, Role: FOLLOWER
I20260808 15:54:57.065420 30291 consensus_queue.cc:260] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.065917 30301 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.066166 30301 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.066464 30290 tablet_bootstrap.cc:492] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: No bootstrap required, opened a new log
I20260808 15:54:57.066617 30290 ts_tablet_manager.cc:1397] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260808 15:54:57.067399 30290 raft_consensus.cc:348] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.067647 30290 raft_consensus.cc:374] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:57.067754 30290 raft_consensus.cc:729] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276d84f70d014349ab9f78a11a6360ae, State: Initialized, Role: FOLLOWER
I20260808 15:54:57.067989 30290 consensus_queue.cc:260] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.068125 30292 ts_tablet_manager.cc:1428] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260808 15:54:57.068851 30296 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.069147 30296 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.069489 30291 ts_tablet_manager.cc:1428] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260808 15:54:57.070526 30290 ts_tablet_manager.cc:1428] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260808 15:54:57.274282 30301 raft_consensus.cc:482] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:54:57.274524 30301 raft_consensus.cc:504] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.275241 30301 leader_election.cc:288] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345), 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757)
I20260808 15:54:57.275756 29944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8d59cde4da443c8b6572022d4562c0" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a085108af47299a03ef7177ef412c" is_pre_election: true
I20260808 15:54:57.275830 30220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8d59cde4da443c8b6572022d4562c0" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276d84f70d014349ab9f78a11a6360ae" is_pre_election: true
I20260808 15:54:57.276059 30220 raft_consensus.cc:2454] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 0.
I20260808 15:54:57.276057 29944 raft_consensus.cc:2454] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 0.
I20260808 15:54:57.276503 30013 leader_election.cc:302] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1957e14960af42a29ade2da06193ac40, 276d84f70d014349ab9f78a11a6360ae, fe0a085108af47299a03ef7177ef412c; no voters:
I20260808 15:54:57.276784 30301 raft_consensus.cc:2781] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:54:57.276903 30301 raft_consensus.cc:482] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:54:57.276979 30301 raft_consensus.cc:3037] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:57.278993 30301 raft_consensus.cc:504] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.279501 30301 leader_election.cc:288] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 election: Requested vote from peers fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345), 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757)
I20260808 15:54:57.279891 29944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8d59cde4da443c8b6572022d4562c0" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a085108af47299a03ef7177ef412c"
I20260808 15:54:57.279960 30220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e8d59cde4da443c8b6572022d4562c0" candidate_uuid: "1957e14960af42a29ade2da06193ac40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276d84f70d014349ab9f78a11a6360ae"
I20260808 15:54:57.280113 29944 raft_consensus.cc:3037] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:57.280172 30220 raft_consensus.cc:3037] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:57.282335 29944 raft_consensus.cc:2454] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 1.
I20260808 15:54:57.282352 30220 raft_consensus.cc:2454] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1957e14960af42a29ade2da06193ac40 in term 1.
I20260808 15:54:57.282758 30014 leader_election.cc:302] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1957e14960af42a29ade2da06193ac40, 276d84f70d014349ab9f78a11a6360ae; no voters:
I20260808 15:54:57.283062 30301 raft_consensus.cc:2781] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:57.283247 30301 raft_consensus.cc:686] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [term 1 LEADER]: Becoming Leader. State: Replica: 1957e14960af42a29ade2da06193ac40, State: Running, Role: LEADER
I20260808 15:54:57.283526 30301 consensus_queue.cc:237] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } }
I20260808 15:54:57.284992 30298 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: TxnStatusTablet state changed. Reason: New leader 1957e14960af42a29ade2da06193ac40. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.285197 30298 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:54:57.285801 30330 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:54:57.286075 30330 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:54:57.286527 29785 catalog_manager.cc:5495] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1957e14960af42a29ade2da06193ac40 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } health_report { overall_health: UNKNOWN } } }
I20260808 15:54:57.309037 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.303224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:39661 (local address 127.0.0.1:60288)
0808 15:54:57.303450 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:57.303470 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:57.303619 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:57.304217 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:57.304221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:57.304229 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:54:57.304379 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:57.304385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.305881 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.305884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:57.306863 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:57.306874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.307903 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.307907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:57.308036 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.308086 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:54:57.308510 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:54:57.308525 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 15:54:57.308593 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 15:54:57.308778 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:54:57.309074 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.303382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.27.254.2:39661)
0808 15:54:57.303877 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.303883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.303902 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:54:57.303965 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.303970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.304048 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.304179 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:57.304542 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.305766 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.306999 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.307766 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.307888 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.308269 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:57.308414 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:57.308503 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.308646 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.308726 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":270,"threads_started":1}
I20260808 15:54:57.313561 30220 raft_consensus.cc:1261] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Refusing update from remote peer 1957e14960af42a29ade2da06193ac40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:57.313561 29944 raft_consensus.cc:1261] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Refusing update from remote peer 1957e14960af42a29ade2da06193ac40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:57.314150 30301 consensus_queue.cc:1035] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:57.314512 30298 consensus_queue.cc:1035] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:57.317713 30297 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: TxnStatusTablet state changed. Reason: New leader 1957e14960af42a29ade2da06193ac40. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.317974 30297 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.321641 30297 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.321888 30297 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P fe0a085108af47299a03ef7177ef412c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.325168 30298 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.326763 30298 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:54:57.327713 30334 mvcc.cc:205] Tried to move back new op lower bound from 7316293620990464000 to 7316293620875436032. Current Snapshot: MvccSnapshot[applied={T|T < 7316293620990464000 or (T in {7316293620990464000})}]
I20260808 15:54:57.334985 30296 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: TxnStatusTablet state changed. Reason: New leader 1957e14960af42a29ade2da06193ac40. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.335273 30296 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.336323 30301 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.336625 30301 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 1957e14960af42a29ade2da06193ac40: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:54:57.341002 30296 tablet_replica.cc:439] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1957e14960af42a29ade2da06193ac40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 } } peers { permanent_uuid: "1957e14960af42a29ade2da06193ac40" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 39661 } } peers { permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 } } }
I20260808 15:54:57.341261 30296 tablet_replica.cc:442] T 8e8d59cde4da443c8b6572022d4562c0 P 276d84f70d014349ab9f78a11a6360ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:54:57.344754 30298 consensus_queue.cc:1035] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "276d84f70d014349ab9f78a11a6360ae" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:57.351959 30298 consensus_queue.cc:1035] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a085108af47299a03ef7177ef412c" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:57.354833 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.346543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:48557 (local address 127.27.254.2:39661)
0808 15:54:57.346742 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.346748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.346938 (+ 190us) server_negotiation.cc:407] Connection header received
0808 15:54:57.347879 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.347887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.347970 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.348033 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:57.348668 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.349695 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.352281 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.353003 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.353101 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.354112 (+ 1011us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:57.354233 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:57.354297 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.354471 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.354539 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:54:57.354952 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.346190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:39661 (local address 127.27.254.2:48557)
0808 15:54:57.346853 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:57.346870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:57.347215 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:57.348233 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:57.348236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:57.348247 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:54:57.348498 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:57.348505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.349976 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.349984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:57.351922 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:57.351936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.353618 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.353627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:57.353819 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.353904 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:54:57.354362 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:54:57.354369 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:54:57.354453 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:54:57.354577 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":572,"threads_started":1}
I20260808 15:54:57.413385 29786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:54:57.413746 29786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:54:57.416851 29786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1f15e68ca4a4ffea186e3d75bb0c4cf: Sending DeleteTablet for 3 replicas of tablet 6765292c71c14143b58cfcaf11a70932
I20260808 15:54:57.417677 30061 tablet_service.cc:1514] Processing DeleteTablet for tablet 6765292c71c14143b58cfcaf11a70932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:54:57 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:60268
I20260808 15:54:57.417702 29922 tablet_service.cc:1514] Processing DeleteTablet for tablet 6765292c71c14143b58cfcaf11a70932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:54:57 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:35170
I20260808 15:54:57.417791 30201 tablet_service.cc:1514] Processing DeleteTablet for tablet 6765292c71c14143b58cfcaf11a70932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:54:57 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:36296
I20260808 15:54:57.429313 30350 tablet_replica.cc:330] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: stopping tablet replica
I20260808 15:54:57.429581 30350 raft_consensus.cc:2229] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:54:57.429862 30350 raft_consensus.cc:2258] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:54:57.432508 30350 ts_tablet_manager.cc:1905] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:54:57.433557 30349 tablet_replica.cc:330] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: stopping tablet replica
I20260808 15:54:57.433826 30349 raft_consensus.cc:2229] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:54:57.434266 30349 raft_consensus.cc:2258] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:54:57.434449 30348 tablet_replica.cc:330] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: stopping tablet replica
I20260808 15:54:57.434737 30348 raft_consensus.cc:2229] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:54:57.435051 30348 raft_consensus.cc:2258] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:54:57.435879 30349 ts_tablet_manager.cc:1905] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:54:57.442734 30350 ts_tablet_manager.cc:1918] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:54:57.443437 30350 log.cc:1196] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/6765292c71c14143b58cfcaf11a70932
I20260808 15:54:57.443636 30349 ts_tablet_manager.cc:1918] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:54:57.443802 30349 log.cc:1196] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/6765292c71c14143b58cfcaf11a70932
I20260808 15:54:57.444265 30350 ts_tablet_manager.cc:1939] T 6765292c71c14143b58cfcaf11a70932 P 276d84f70d014349ab9f78a11a6360ae: Deleting consensus metadata
I20260808 15:54:57.444429 30349 ts_tablet_manager.cc:1939] T 6765292c71c14143b58cfcaf11a70932 P 1957e14960af42a29ade2da06193ac40: Deleting consensus metadata
I20260808 15:54:57.445542 29771 catalog_manager.cc:4841] TS 276d84f70d014349ab9f78a11a6360ae (127.27.254.3:45757): tablet 6765292c71c14143b58cfcaf11a70932 (table test-table [id=f56e3cc4f3f14b47a565f246b7c7f6bb]) successfully deleted
I20260808 15:54:57.445542 29772 catalog_manager.cc:4841] TS 1957e14960af42a29ade2da06193ac40 (127.27.254.2:39661): tablet 6765292c71c14143b58cfcaf11a70932 (table test-table [id=f56e3cc4f3f14b47a565f246b7c7f6bb]) successfully deleted
I20260808 15:54:57.446177 30348 ts_tablet_manager.cc:1905] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:54:57.453039 30348 ts_tablet_manager.cc:1918] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:54:57.453174 30348 log.cc:1196] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/6765292c71c14143b58cfcaf11a70932
I20260808 15:54:57.453714 30348 ts_tablet_manager.cc:1939] T 6765292c71c14143b58cfcaf11a70932 P fe0a085108af47299a03ef7177ef412c: Deleting consensus metadata
I20260808 15:54:57.455168 29770 catalog_manager.cc:4841] TS fe0a085108af47299a03ef7177ef412c (127.27.254.1:44345): tablet 6765292c71c14143b58cfcaf11a70932 (table test-table [id=f56e3cc4f3f14b47a565f246b7c7f6bb]) successfully deleted
I20260808 15:54:57.459964 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.430930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.27.254.1:44345)
0808 15:54:57.444127 (+ 13197us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.444134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.444160 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:54:57.444222 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.444227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.444306 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.444469 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:57.446514 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.447341 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.449188 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.449721 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.449865 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.454019 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:57.454033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:57.454048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:57.454077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:57.457074 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.457840 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.457848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.457850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.458002 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.458396 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.458400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.458402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.458716 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:57.459018 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.459309 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.459530 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12522,"thread_start_us":12350,"threads_started":1}
W20260808 15:54:57.460659 29988 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35180
I20260808 15:54:57.476563 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.463511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.27.254.62:42975)
0808 15:54:57.463765 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.463774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.464224 (+ 450us) server_negotiation.cc:407] Connection header received
0808 15:54:57.464485 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.464492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.464574 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.464691 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:57.465967 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.467100 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.468269 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.468906 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.469026 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.471302 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:57.471310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:57.471319 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:57.471350 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:57.473999 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.474636 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.474643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.474646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.474771 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.475136 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.475141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.475146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.475422 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:57.475695 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.475993 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.476219 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 15:54:57.477316 29786 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48874
I20260808 15:54:57.479072 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29857
I20260808 15:54:57.505555 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.480186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.27.254.1:44345)
0808 15:54:57.480353 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.480359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.489472 (+ 9113us) server_negotiation.cc:407] Connection header received
0808 15:54:57.489900 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.489906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.489989 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.490056 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:57.492022 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.493105 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.495066 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.495786 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.495882 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.499002 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:57.499009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:57.499014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:57.499050 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:57.502616 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.503425 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.503431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.503435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.503563 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.503940 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.503945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.503949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.504252 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:57.504467 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.504950 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.505163 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 15:54:57.506944 29988 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184
W20260808 15:54:57.507608 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29857: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:57.507840 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29857
I20260808 15:54:57.525017 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29997
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:54:57.540414 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.526260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.27.254.2:39661)
0808 15:54:57.526441 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.526447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.526474 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:54:57.526525 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.526528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.526601 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.526715 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:57.527827 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.528629 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.529925 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.530411 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.530545 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.534543 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:57.534557 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:57.534575 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:57.534614 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:57.537645 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.538293 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.538301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.538303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.538440 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.538815 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.538821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.538824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.539193 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:57.539513 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.539802 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.540027 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:54:57.541157 30127 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60304
W20260808 15:54:57.541987 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:57.542166 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29997
I20260808 15:54:57.560364 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30136
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[29741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:54:57.575613 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:57.561143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.27.254.3:45757)
0808 15:54:57.561709 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:57.561713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:57.561729 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:54:57.561794 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:57.561798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:57.561859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:57.561951 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:57.563125 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.563873 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.565042 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:57.565561 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:57.565676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:57.569891 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:57.569906 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:57.569922 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:57.569964 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:57.573040 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.573725 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.573733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.573735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.573877 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:57.574202 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:57.574205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:57.574207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:57.574551 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:57.574794 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:57.575058 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:54:57.575243 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
W20260808 15:54:57.576303 30266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36304
W20260808 15:54:57.576854 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30136: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:57.577003 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 30136
I20260808 15:54:57.594895 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 29754
I20260808 15:54:57.595999 29826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:57.712324 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 29754
2026-08-08T15:54:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4536 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[30361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[30361](info): set up 2 sockets
Aug 08 15:54:57 dist-test-slave-srgv krb5kdc[30361](info): commencing operation
krb5kdc: starting...
W20260808 15:54:58.779578 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:54:58 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:54:58.967238 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37771
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:59.290324 30374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:59.290712 30374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:59.290875 30374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:59.298141 30374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:59.298309 30374 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:54:59.298421 30374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:59.298533 30374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:59.298647 30374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:59.298743 30374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:59.308776 30374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37771
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:59.310220 30374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:59.311551 30374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:59.323203 30382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:59.323292 30380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:59.323294 30379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:59.324060 30374 server_base.cc:1034] running on GCE node
I20260808 15:54:59.324893 30374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:59.326440 30374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:59.327693 30374 hybrid_clock.cc:648] HybridClock initialized: now 1786204499327635 us; error 38 us; skew 500 ppm
I20260808 15:54:59.328022 30374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:59 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204499, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:59.331204 30374 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:54:59.332679 30374 webserver.cc:460] Webserver started at http://127.27.254.62:41299/ using document root <none> and password file <none>
I20260808 15:54:59.333348 30374 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:59.333494 30374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:59.333781 30374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:59.336637 30374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "b66160b82eec44518558c99eba80a0ce"
format_stamp: "Formatted at 2026-08-08 15:54:59 on dist-test-slave-srgv"
server_key: "8e4c229d752eded64267b44b59d03bd2"
server_key_iv: "e2efb8d8033d0a93681363f9400867e5"
server_key_version: "encryptionkey@0"
I20260808 15:54:59.337347 30374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "b66160b82eec44518558c99eba80a0ce"
format_stamp: "Formatted at 2026-08-08 15:54:59 on dist-test-slave-srgv"
server_key: "8e4c229d752eded64267b44b59d03bd2"
server_key_iv: "e2efb8d8033d0a93681363f9400867e5"
server_key_version: "encryptionkey@0"
I20260808 15:54:59.342566 30374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:54:59.346058 30389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:59.347364 30374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 15:54:59.347543 30374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "b66160b82eec44518558c99eba80a0ce"
format_stamp: "Formatted at 2026-08-08 15:54:59 on dist-test-slave-srgv"
server_key: "8e4c229d752eded64267b44b59d03bd2"
server_key_iv: "e2efb8d8033d0a93681363f9400867e5"
server_key_version: "encryptionkey@0"
I20260808 15:54:59.347772 30374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:59.365937 30374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:59.370347 30374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:59.370640 30374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:59.392446 30374 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:37771
I20260808 15:54:59.392472 30450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:37771 every 8 connection(s)
I20260808 15:54:59.393890 30374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:59.395123 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 30374
I20260808 15:54:59.395339 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:54:59.396130 28664 external_mini_cluster.cc:1428] Setting key a46608b75f04f4fc684d9e6173fa11f8
I20260808 15:54:59.398854 30451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:54:59 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:59.411824 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Bootstrap starting.
I20260808 15:54:59.416220 30451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:59.417534 30451 log.cc:824] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:59.420998 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: No bootstrap required, opened a new log
I20260808 15:54:59.421020 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:59.398306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.27.254.62:37771)
0808 15:54:59.399122 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:59.399135 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:59.399182 (+ 47us) server_negotiation.cc:407] Connection header received
0808 15:54:59.400884 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:59.400906 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:59.401331 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:59.401765 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:59.403501 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:59.404848 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:59.406609 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:59.407252 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:59.407380 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:59.412054 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:59.412063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:59.412081 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:59.412130 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:59.416184 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:59.416980 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:59.416990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:59.416998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:59.417226 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:59.417670 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:59.417676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:59.417679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:59.418383 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:59.418612 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:59.419240 (+ 628us) server_negotiation.cc:299] Negotiation successful
0808 15:54:59.419663 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":384,"threads_started":1}
I20260808 15:54:59.429008 30451 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66160b82eec44518558c99eba80a0ce" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37771 } }
I20260808 15:54:59.429347 30451 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:59.429458 30451 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66160b82eec44518558c99eba80a0ce, State: Initialized, Role: FOLLOWER
I20260808 15:54:59.430050 30451 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66160b82eec44518558c99eba80a0ce" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37771 } }
I20260808 15:54:59.430270 30451 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:59.430366 30451 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:59.430491 30451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:59.432884 30451 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66160b82eec44518558c99eba80a0ce" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37771 } }
I20260808 15:54:59.433368 30451 leader_election.cc:302] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66160b82eec44518558c99eba80a0ce; no voters:
I20260808 15:54:59.434005 30451 leader_election.cc:288] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:59.434218 30456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:59.434944 30456 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [term 1 LEADER]: Becoming Leader. State: Replica: b66160b82eec44518558c99eba80a0ce, State: Running, Role: LEADER
I20260808 15:54:59.435473 30456 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66160b82eec44518558c99eba80a0ce" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37771 } }
I20260808 15:54:59.436157 30451 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:59.438426 30457 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66160b82eec44518558c99eba80a0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66160b82eec44518558c99eba80a0ce" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37771 } } }
I20260808 15:54:59.438241 30458 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66160b82eec44518558c99eba80a0ce. Latest consensus state: current_term: 1 leader_uuid: "b66160b82eec44518558c99eba80a0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66160b82eec44518558c99eba80a0ce" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37771 } } }
I20260808 15:54:59.438908 30457 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:59.438948 30458 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:59.442028 30463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:59.447533 30463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:59.456439 30463 catalog_manager.cc:1347] Generated new cluster ID: c4e89ff4b73b49c6af5745b99a3f029f
I20260808 15:54:59.456594 30463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:59.491155 30463 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:59.492486 30463 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:59.502883 30463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Generated new TSK 0
I20260808 15:54:59.503703 30463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:54:59.555055 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37771
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:59.886234 30477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:59.886529 30477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:59.886709 30477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:59.894105 30477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:59.894250 30477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:59.894419 30477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:54:59.894501 30477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:54:59.894634 30477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:54:59.904660 30477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:59.906186 30477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:59.907579 30477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:59.921141 30485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:59.921283 30482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:59.921379 30483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:59.922847 30477 server_base.cc:1034] running on GCE node
I20260808 15:54:59.923537 30477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:59.924540 30477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:59.925765 30477 hybrid_clock.cc:648] HybridClock initialized: now 1786204499925722 us; error 65 us; skew 500 ppm
I20260808 15:54:59.926097 30477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:59 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204499, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:59.929126 30477 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:54:59.930714 30477 webserver.cc:460] Webserver started at http://127.27.254.1:38901/ using document root <none> and password file <none>
I20260808 15:54:59.931421 30477 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:59.931569 30477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:59.931895 30477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:59.934949 30477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "7bef63b80d9449068e40d8da73d4afc9"
format_stamp: "Formatted at 2026-08-08 15:54:59 on dist-test-slave-srgv"
server_key: "501f348bc9b42df941c0f62164296e73"
server_key_iv: "804266b5d1b638660268de20dd0506b1"
server_key_version: "encryptionkey@0"
I20260808 15:54:59.935740 30477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "7bef63b80d9449068e40d8da73d4afc9"
format_stamp: "Formatted at 2026-08-08 15:54:59 on dist-test-slave-srgv"
server_key: "501f348bc9b42df941c0f62164296e73"
server_key_iv: "804266b5d1b638660268de20dd0506b1"
server_key_version: "encryptionkey@0"
I20260808 15:54:59.941038 30477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:54:59.946070 30492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:59.947995 30477 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:54:59.948450 30477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "7bef63b80d9449068e40d8da73d4afc9"
format_stamp: "Formatted at 2026-08-08 15:54:59 on dist-test-slave-srgv"
server_key: "501f348bc9b42df941c0f62164296e73"
server_key_iv: "804266b5d1b638660268de20dd0506b1"
server_key_version: "encryptionkey@0"
I20260808 15:54:59.948782 30477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:59.964110 30477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:59.968192 30477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:59.968502 30477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:59.971226 30477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:54:59.971392 30477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:59.971524 30477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:54:59.971625 30477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:54:59 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204499, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:54:59.996897 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:59.972014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:60201 (local address 127.27.254.62:37771)
0808 15:54:59.972583 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:59.972589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:59.974006 (+ 1417us) server_negotiation.cc:407] Connection header received
0808 15:54:59.976204 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:59.976211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:59.976354 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:59.976525 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:59.979261 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:59.980309 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:59.983594 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:59.984121 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:59.984277 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:59.990355 (+ 6078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:59.990360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:59.990363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:59.990398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:59.993050 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:59.994303 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:59.994311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:59.994316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:59.994457 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:59.994944 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:59.994950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:59.994955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:59.995363 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:59.995526 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:59.996284 (+ 758us) server_negotiation.cc:299] Negotiation successful
0808 15:54:59.996512 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":323,"threads_started":1}
I20260808 15:54:59.998342 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:59.972743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.27.254.1:60201)
0808 15:54:59.973805 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:59.973867 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:59.975884 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:59.976851 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:59.976867 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:59.977877 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:59.979021 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:59.979058 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:59.980494 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:59.980505 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:59.983408 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:59.983428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:59.984454 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:59.984466 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:59.984841 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:59.985747 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:59.985791 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:59.990031 (+ 4240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:59.993212 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:59.993228 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:59.993252 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:59.994136 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:59.994585 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:59.994590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:59.994594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:59.994809 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:59.995510 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:59.995519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:59.995909 (+ 390us) client_negotiation.cc:769] Sending connection context
0808 15:54:59.996181 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 15:54:59.996750 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":446,"threads_started":1}
I20260808 15:55:00.020699 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:00.006312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.27.254.1:47987)
0808 15:55:00.006925 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:00.006945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:00.007121 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:00.007467 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:00.007471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:00.007939 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:00.008223 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:00.008232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.009235 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.009240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:00.010868 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:00.010884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.011861 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.011867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:00.012121 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:00.012961 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:00.012992 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:00.014189 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:00.018046 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:00.018049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:00.018052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:00.018481 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:00.018961 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:00.018966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:00.018970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:00.019112 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:00.019834 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:00.019837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:00.020001 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 15:55:00.020112 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 15:55:00.020372 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":380,"threads_started":1}
I20260808 15:55:00.021077 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:00.006207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:47987 (local address 127.27.254.62:37771)
0808 15:55:00.006685 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:00.006692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:00.007043 (+ 351us) server_negotiation.cc:407] Connection header received
0808 15:55:00.007255 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:00.007260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:00.007337 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:00.007441 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:00.008369 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.009086 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.011018 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.011658 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.011808 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:00.014436 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:00.014445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:00.014453 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:00.014505 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:00.017909 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:00.018613 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:00.018621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:00.018627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:00.018824 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:00.019230 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:00.019238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:00.019244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:00.019708 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:00.019908 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:00.020282 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 15:55:00.020590 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260808 15:55:00.030669 30477 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:37653
I20260808 15:55:00.030707 30612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:37653 every 8 connection(s)
I20260808 15:55:00.031982 30477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:00.034271 30613 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37771
I20260808 15:55:00.034505 30613 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:00.034512 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 30477
I20260808 15:55:00.034721 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:00.035254 30613 heartbeater.cc:511] Master 127.27.254.62:37771 requested a full tablet report, sending...
I20260808 15:55:00.035396 28664 external_mini_cluster.cc:1428] Setting key 7a351ea1e39e07d36beadc0b4e034459
I20260808 15:55:00.037547 30406 ts_manager.cc:194] Registered new tserver with Master: 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653)
I20260808 15:55:00.038935 30406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:60201
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:00.076967 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37771
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:00.409155 30617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:00.409556 30617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:00.409751 30617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:00.417191 30617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:00.417306 30617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:00.417471 30617 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:55:00.417547 30617 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:55:00.417657 30617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:00.427675 30617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:00.429054 30617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:00.430474 30617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:00.444721 30622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:00.444756 30625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:00.444732 30623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:00.445493 30617 server_base.cc:1034] running on GCE node
I20260808 15:55:00.446228 30617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:00.447237 30617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:00.448441 30617 hybrid_clock.cc:648] HybridClock initialized: now 1786204500448377 us; error 59 us; skew 500 ppm
I20260808 15:55:00.448798 30617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:00 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204500, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:00.452134 30617 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:00.453830 30617 webserver.cc:460] Webserver started at http://127.27.254.2:42621/ using document root <none> and password file <none>
I20260808 15:55:00.454638 30617 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:00.454792 30617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:00.455127 30617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:00.458280 30617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "1a87e996db5b48b9b545fc1a379e26f3"
format_stamp: "Formatted at 2026-08-08 15:55:00 on dist-test-slave-srgv"
server_key: "bcaf2c177d62d67349d1b7f5e124f86c"
server_key_iv: "1b58cdf4de470d60fa95ec012dee67e9"
server_key_version: "encryptionkey@0"
I20260808 15:55:00.459092 30617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "1a87e996db5b48b9b545fc1a379e26f3"
format_stamp: "Formatted at 2026-08-08 15:55:00 on dist-test-slave-srgv"
server_key: "bcaf2c177d62d67349d1b7f5e124f86c"
server_key_iv: "1b58cdf4de470d60fa95ec012dee67e9"
server_key_version: "encryptionkey@0"
I20260808 15:55:00.464740 30617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:55:00.468498 30632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:00.470034 30617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:00.470252 30617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "1a87e996db5b48b9b545fc1a379e26f3"
format_stamp: "Formatted at 2026-08-08 15:55:00 on dist-test-slave-srgv"
server_key: "bcaf2c177d62d67349d1b7f5e124f86c"
server_key_iv: "1b58cdf4de470d60fa95ec012dee67e9"
server_key_version: "encryptionkey@0"
I20260808 15:55:00.470484 30617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:00.496529 30617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:00.500845 30617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:00.501175 30617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:00.504236 30617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:00.504369 30617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:00.504504 30617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:00.504604 30617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:55:00 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204500, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:00.530326 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:00.504800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44683 (local address 127.27.254.62:37771)
0808 15:55:00.505005 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:00.505013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:00.506714 (+ 1701us) server_negotiation.cc:407] Connection header received
0808 15:55:00.508901 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:00.508907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:00.509008 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:00.509151 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:00.511697 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.512799 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.515595 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.516249 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.516420 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:00.522414 (+ 5994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:00.522420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:00.522423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:00.522459 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:00.526236 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:00.527343 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:00.527352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:00.527358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:00.527556 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:00.528056 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:00.528061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:00.528065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:00.528373 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:00.528571 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:00.529852 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0808 15:55:00.530046 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 15:55:00.532430 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:00.505592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.27.254.2:44683)
0808 15:55:00.506502 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:00.506551 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:00.508485 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:00.509511 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:00.509525 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:00.510411 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:00.511452 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:00.511486 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.512970 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.512979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:00.515436 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:00.515449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.516592 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.516605 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:00.517017 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:00.518054 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:00.518095 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:00.522080 (+ 3985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:00.526439 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:00.526454 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:00.526477 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:00.527164 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:00.527692 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:00.527695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:00.527698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:00.527914 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:00.528562 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:00.528574 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:00.529253 (+ 679us) client_negotiation.cc:769] Sending connection context
0808 15:55:00.529714 (+ 461us) client_negotiation.cc:240] Negotiation successful
0808 15:55:00.530504 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":411,"threads_started":1}
I20260808 15:55:00.555655 30617 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:40511
I20260808 15:55:00.555684 30751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:40511 every 8 connection(s)
I20260808 15:55:00.556246 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:00.539948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:45749 (local address 127.27.254.62:37771)
0808 15:55:00.540193 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:00.540202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:00.540228 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:55:00.540308 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:00.540316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:00.540418 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:00.540549 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:00.541878 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.543055 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.545295 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.546133 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.546307 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:00.548650 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:00.548655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:00.548659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:00.548692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:00.552754 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:00.553683 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:00.553688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:00.553692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:00.553859 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:00.554370 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:00.554379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:00.554385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:00.554928 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:00.555112 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:00.555666 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 15:55:00.555901 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:55:00.556383 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:00.539322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.27.254.2:45749)
0808 15:55:00.540001 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:00.540035 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:00.540158 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:00.540588 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:00.540591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:00.541052 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:00.541645 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:00.541667 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.543292 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.543300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:00.545034 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:00.545050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:00.546370 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:00.546379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:00.546563 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:00.547254 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:00.547272 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:00.548452 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:00.552905 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:00.552914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:00.552921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:00.553549 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:00.553978 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:00.553982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:00.553985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:00.554194 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:00.555085 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:00.555097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:00.555356 (+ 259us) client_negotiation.cc:769] Sending connection context
0808 15:55:00.555553 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 15:55:00.555932 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":373,"threads_started":1}
I20260808 15:55:00.557135 30617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:00.559628 30752 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37771
I20260808 15:55:00.559900 30752 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:00.560576 30752 heartbeater.cc:511] Master 127.27.254.62:37771 requested a full tablet report, sending...
I20260808 15:55:00.561959 30406 ts_manager.cc:194] Registered new tserver with Master: 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511)
I20260808 15:55:00.562772 30406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:44683
I20260808 15:55:00.566944 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 30617
I20260808 15:55:00.567152 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:00.567595 28664 external_mini_cluster.cc:1428] Setting key 9685063d5748fc5963fb9ddfcb0ed246
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:00.609833 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37771
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:00.947248 30756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:00.947551 30756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:00.947734 30756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:00.955654 30756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:00.955801 30756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:00.955940 30756 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:55:00.956024 30756 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:55:00.956149 30756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:00.966369 30756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:00.967646 30756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:00.969045 30756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:00.983071 30761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:00.983016 30762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:00.983081 30764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:00.984009 30756 server_base.cc:1034] running on GCE node
I20260808 15:55:00.984730 30756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:00.985843 30756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:00.987074 30756 hybrid_clock.cc:648] HybridClock initialized: now 1786204500987022 us; error 49 us; skew 500 ppm
I20260808 15:55:00.987417 30756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:00 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204500, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:00.990737 30756 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:00.992436 30756 webserver.cc:460] Webserver started at http://127.27.254.3:37209/ using document root <none> and password file <none>
I20260808 15:55:00.993181 30756 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:00.993345 30756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:00.993723 30756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:00.996783 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "616cba7de7104feb99646f05b2167b14"
format_stamp: "Formatted at 2026-08-08 15:55:00 on dist-test-slave-srgv"
server_key: "283210be9c19e1600047f454fffaa154"
server_key_iv: "4158e051adfc9137c4055d5dd0feea36"
server_key_version: "encryptionkey@0"
I20260808 15:55:00.997730 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "616cba7de7104feb99646f05b2167b14"
format_stamp: "Formatted at 2026-08-08 15:55:00 on dist-test-slave-srgv"
server_key: "283210be9c19e1600047f454fffaa154"
server_key_iv: "4158e051adfc9137c4055d5dd0feea36"
server_key_version: "encryptionkey@0"
I20260808 15:55:01.003409 30756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:01.007429 30771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:01.008852 30756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:01.009052 30756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "616cba7de7104feb99646f05b2167b14"
format_stamp: "Formatted at 2026-08-08 15:55:00 on dist-test-slave-srgv"
server_key: "283210be9c19e1600047f454fffaa154"
server_key_iv: "4158e051adfc9137c4055d5dd0feea36"
server_key_version: "encryptionkey@0"
I20260808 15:55:01.009287 30756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:01.028330 30756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:01.033118 30756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:01.033532 30756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:01.036897 30756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:01.037020 30756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:01.037195 30756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:01.037282 30756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:01.041832 30613 heartbeater.cc:503] Master 127.27.254.62:37771 was elected leader, sending a full tablet report...
Aug 08 15:55:01 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204500, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:01.063449 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.037628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:43995 (local address 127.27.254.62:37771)
0808 15:55:01.037863 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.037869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.039430 (+ 1561us) server_negotiation.cc:407] Connection header received
0808 15:55:01.042637 (+ 3207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.042643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.042743 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.042877 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:01.045519 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.046285 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.049559 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.050142 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.050298 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.056429 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:01.056435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:01.056438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:01.056475 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:01.059725 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.060518 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.060524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.060527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.060677 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.061153 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.061158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.061162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.061549 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:01.061762 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.062987 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.063174 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 15:55:01.065593 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.038297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.27.254.3:43995)
0808 15:55:01.039210 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.039273 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.042263 (+ 2990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.043188 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.043205 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.044188 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:01.045242 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.045274 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.046451 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.046459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.049301 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.049322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.050443 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.050455 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.050920 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.051889 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:01.051948 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:01.056127 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:01.059887 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:01.059897 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:01.059915 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:01.060370 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:01.060804 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:01.060809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:01.060813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:01.061023 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:01.061740 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:01.061752 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:01.062381 (+ 629us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.062799 (+ 418us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.063579 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":351,"threads_started":1}
I20260808 15:55:01.090222 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.073013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:44611 (local address 127.27.254.62:37771)
0808 15:55:01.073258 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.073268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.074200 (+ 932us) server_negotiation.cc:407] Connection header received
0808 15:55:01.074348 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.074356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.074457 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.074602 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:01.075968 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.077286 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.079577 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.080014 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.080153 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.083009 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:01.083018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:01.083026 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:01.083073 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:01.087029 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.088065 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.088073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.088078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.088260 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.088659 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.088665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.088670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.088952 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:01.089123 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.089579 (+ 456us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.089819 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:55:01.090337 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.073017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.27.254.3:44611)
0808 15:55:01.074070 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.074091 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.074235 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.074634 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.074639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.075358 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:01.075760 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.075777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.077486 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.077494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.079426 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.079439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.080236 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.080243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.080436 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.081249 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:01.081269 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:01.082697 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:01.087183 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:01.087190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:01.087194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:01.087855 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:01.088407 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:01.088411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:01.088413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:01.088527 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:01.089072 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:01.089077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:01.089266 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.089403 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.089857 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":756,"threads_started":1}
I20260808 15:55:01.101982 30756 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:38107
I20260808 15:55:01.102031 30890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:38107 every 8 connection(s)
I20260808 15:55:01.103299 30756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:01.105559 30891 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37771
I20260808 15:55:01.105772 30891 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:01.106393 30891 heartbeater.cc:511] Master 127.27.254.62:37771 requested a full tablet report, sending...
I20260808 15:55:01.107771 30406 ts_manager.cc:194] Registered new tserver with Master: 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:01.108577 30406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:43995
I20260808 15:55:01.109195 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 30756
I20260808 15:55:01.109385 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:01.109875 28664 external_mini_cluster.cc:1428] Setting key 02183a94b633cb4a2a6dde7ed5d08b7e
I20260808 15:55:01.112769 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:55:01 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:55:01 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:01.162115 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.147179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.27.254.62:37771)
0808 15:55:01.147439 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.147446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.147726 (+ 280us) server_negotiation.cc:407] Connection header received
0808 15:55:01.147883 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.147887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.147954 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.148045 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:01.149569 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.150359 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.151803 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.152339 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.152501 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.156671 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:01.156676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:01.156679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:01.156726 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:01.159422 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.160063 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.160067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.160069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.160201 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.160654 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.160658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.160660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.160968 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:01.161187 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.161551 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.161745 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 15:55:01.166433 30406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:01.173668 30406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:01.208494 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.188056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.27.254.3:38107)
0808 15:55:01.199564 (+ 11508us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.199577 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.200557 (+ 980us) server_negotiation.cc:407] Connection header received
0808 15:55:01.200681 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.200691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.200967 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.201178 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:01.202418 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.203910 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.206329 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.207286 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.207450 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.207707 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.207979 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.208145 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11251,"thread_start_us":12230,"threads_started":1}
I20260808 15:55:01.208500 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.187070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:38107 (local address 127.0.0.1:33352)
0808 15:55:01.200393 (+ 13323us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.200412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.200542 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.201523 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.201528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.201554 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:01.202201 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.202216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.204066 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.204071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.206174 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.206184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.207503 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.207509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.207725 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.207786 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.207923 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.208126 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13198,"thread_start_us":11062,"threads_started":1}
I20260808 15:55:01.209048 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.186535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40511 (local address 127.0.0.1:53072)
0808 15:55:01.197546 (+ 11011us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.197613 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.197873 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.200732 (+ 2859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.200739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.200772 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:01.201198 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.201214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.203021 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.203030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.205765 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.205786 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.207958 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.207966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.208155 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.208187 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.208289 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.208426 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10577,"mutex_wait_us":42,"thread_start_us":13596,"threads_started":1}
I20260808 15:55:01.209048 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.186733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.27.254.2:40511)
0808 15:55:01.200005 (+ 13272us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.200015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.200051 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:55:01.200176 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.200185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.200492 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.200697 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:01.201497 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.202893 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.205966 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.207413 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.207562 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.207790 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.208607 (+ 817us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.208785 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13126,"thread_start_us":13015,"threads_started":1}
I20260808 15:55:01.211215 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.200850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.27.254.1:37653)
0808 15:55:01.201540 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.201548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.202523 (+ 975us) server_negotiation.cc:407] Connection header received
0808 15:55:01.202668 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.202676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.202920 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.203126 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:01.203854 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.205256 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.207772 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.209266 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.209413 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.209642 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.210728 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.210907 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
I20260808 15:55:01.212085 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.201639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37653 (local address 127.0.0.1:58076)
0808 15:55:01.202388 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.202407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.202526 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.203129 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.203133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.203164 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:01.203641 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.203661 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.205470 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.205475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.207588 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.207602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.210316 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.210324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.210508 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.210538 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.211591 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.211746 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":4,"thread_start_us":453,"threads_started":1}
I20260808 15:55:01.212863 30826 tablet_service.cc:1467] Processing CreateTablet for tablet db5f4917de5d4327a22f3c6fe7574402 (DEFAULT_TABLE table=test-table [id=1607894c86bd4a4ab222c22d02fd4966]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:01.212852 30682 tablet_service.cc:1467] Processing CreateTablet for tablet db5f4917de5d4327a22f3c6fe7574402 (DEFAULT_TABLE table=test-table [id=1607894c86bd4a4ab222c22d02fd4966]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:01.214356 30682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5f4917de5d4327a22f3c6fe7574402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:01.214421 30826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5f4917de5d4327a22f3c6fe7574402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:01.215776 30543 tablet_service.cc:1467] Processing CreateTablet for tablet db5f4917de5d4327a22f3c6fe7574402 (DEFAULT_TABLE table=test-table [id=1607894c86bd4a4ab222c22d02fd4966]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:01.217274 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5f4917de5d4327a22f3c6fe7574402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:01.225499 30910 tablet_bootstrap.cc:492] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Bootstrap starting.
I20260808 15:55:01.229125 30911 tablet_bootstrap.cc:492] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Bootstrap starting.
I20260808 15:55:01.229876 30910 tablet_bootstrap.cc:654] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:01.231261 30910 log.cc:824] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:01.232295 30912 tablet_bootstrap.cc:492] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Bootstrap starting.
I20260808 15:55:01.233047 30911 tablet_bootstrap.cc:654] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:01.234402 30911 log.cc:824] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:01.236040 30912 tablet_bootstrap.cc:654] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:01.236429 30910 tablet_bootstrap.cc:492] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: No bootstrap required, opened a new log
I20260808 15:55:01.236737 30910 ts_tablet_manager.cc:1397] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260808 15:55:01.237172 30912 log.cc:824] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:01.238123 30911 tablet_bootstrap.cc:492] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: No bootstrap required, opened a new log
I20260808 15:55:01.238415 30911 ts_tablet_manager.cc:1397] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260808 15:55:01.240322 30912 tablet_bootstrap.cc:492] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: No bootstrap required, opened a new log
I20260808 15:55:01.240573 30912 ts_tablet_manager.cc:1397] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:55:01.247754 30912 raft_consensus.cc:348] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.247798 30910 raft_consensus.cc:348] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.248198 30912 raft_consensus.cc:374] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:01.248234 30910 raft_consensus.cc:374] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:01.248294 30912 raft_consensus.cc:729] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Initialized, Role: FOLLOWER
I20260808 15:55:01.248327 30910 raft_consensus.cc:729] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87e996db5b48b9b545fc1a379e26f3, State: Initialized, Role: FOLLOWER
I20260808 15:55:01.248998 30912 consensus_queue.cc:260] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.248977 30910 consensus_queue.cc:260] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.249202 30911 raft_consensus.cc:348] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.249727 30911 raft_consensus.cc:374] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:01.249853 30911 raft_consensus.cc:729] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616cba7de7104feb99646f05b2167b14, State: Initialized, Role: FOLLOWER
I20260808 15:55:01.250504 30911 consensus_queue.cc:260] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.251313 30912 ts_tablet_manager.cc:1428] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260808 15:55:01.251382 30910 ts_tablet_manager.cc:1428] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260808 15:55:01.252243 30752 heartbeater.cc:503] Master 127.27.254.62:37771 was elected leader, sending a full tablet report...
I20260808 15:55:01.253077 30891 heartbeater.cc:503] Master 127.27.254.62:37771 was elected leader, sending a full tablet report...
I20260808 15:55:01.253772 30911 ts_tablet_manager.cc:1428] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260808 15:55:01.284498 30614 tablet.cc:2369] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:55:01.309540 30753 tablet.cc:2369] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:01.346541 30916 raft_consensus.cc:482] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:01.346921 30916 raft_consensus.cc:504] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.348627 30916 leader_election.cc:288] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
W20260808 15:55:01.354969 30892 tablet.cc:2369] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:01.355866 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.349041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:56379 (local address 127.27.254.2:40511)
0808 15:55:01.349309 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.349319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.350168 (+ 849us) server_negotiation.cc:407] Connection header received
0808 15:55:01.350393 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.350398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.350458 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.350555 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:01.351174 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.351939 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.353945 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.354822 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.355021 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.355103 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.355473 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.355598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 15:55:01.356019 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.348878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40511 (local address 127.27.254.1:56379)
0808 15:55:01.350037 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.350053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.350248 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.350568 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.350573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.350595 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:01.351025 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.351037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.352066 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.352073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.353775 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.353787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.355075 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.355082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.355311 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.355337 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.355446 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.355576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":21,"thread_start_us":969,"threads_started":1}
I20260808 15:55:01.356492 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.349071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:37561 (local address 127.27.254.3:38107)
0808 15:55:01.349317 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.349326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.350289 (+ 963us) server_negotiation.cc:407] Connection header received
0808 15:55:01.350484 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.350490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.350554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.350658 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:01.351316 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.352221 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.354124 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.355338 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.355507 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.355571 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.356132 (+ 561us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.356255 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:55:01.356581 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.348946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:38107 (local address 127.27.254.1:37561)
0808 15:55:01.350174 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.350193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.350349 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.350703 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.350709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.350746 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:01.351171 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.351181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.352351 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.352359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.353960 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.353972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.355747 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.355757 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.355934 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.355974 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.356089 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.356210 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":20,"thread_start_us":628,"threads_started":1}
I20260808 15:55:01.357218 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5f4917de5d4327a22f3c6fe7574402" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3" is_pre_election: true
I20260808 15:55:01.357569 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5f4917de5d4327a22f3c6fe7574402" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14" is_pre_election: true
I20260808 15:55:01.357764 30706 raft_consensus.cc:2454] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:01.358076 30846 raft_consensus.cc:2454] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:01.358470 30494 leader_election.cc:302] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:01.358925 30916 raft_consensus.cc:2781] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:01.359081 30916 raft_consensus.cc:482] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:01.359198 30916 raft_consensus.cc:3037] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:01.361387 30916 raft_consensus.cc:504] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.362048 30916 leader_election.cc:288] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:01.362568 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5f4917de5d4327a22f3c6fe7574402" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14"
I20260808 15:55:01.362571 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5f4917de5d4327a22f3c6fe7574402" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3"
I20260808 15:55:01.362782 30846 raft_consensus.cc:3037] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:01.362808 30706 raft_consensus.cc:3037] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:01.364905 30846 raft_consensus.cc:2454] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:01.364943 30706 raft_consensus.cc:2454] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:01.365341 30494 leader_election.cc:302] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:01.365658 30916 raft_consensus.cc:2781] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:01.366128 30916 raft_consensus.cc:686] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Running, Role: LEADER
I20260808 15:55:01.366729 30916 consensus_queue.cc:237] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.371681 30404 catalog_manager.cc:5495] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1). New cstate: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:55:01 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204499, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:01.425765 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.411889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.27.254.62:37771)
0808 15:55:01.412070 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.412075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.412715 (+ 640us) server_negotiation.cc:407] Connection header received
0808 15:55:01.412926 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.412930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.412983 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.413086 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:01.414445 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.415196 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.416453 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.416901 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.417028 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.420357 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:01.420361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:01.420365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:01.420394 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:01.422955 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.423684 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.423691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.423695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.423816 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:01.424203 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:01.424209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:01.424212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:01.424485 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:01.424635 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.425038 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.425384 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":61}
I20260808 15:55:01.425806 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.411904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37771 (local address 127.0.0.1:54876)
0808 15:55:01.412587 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.412604 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.412785 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.413111 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.413124 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.414011 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:01.414294 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.414305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.415323 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.415329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.416302 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.416314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.417092 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.417099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.417254 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.417945 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:01.417967 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:01.420153 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:01.423104 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:01.423121 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:01.423130 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:01.423557 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:01.423931 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:01.423939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:01.423942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:01.424079 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:01.424611 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:01.424619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:01.424791 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.424956 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.425258 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":350,"threads_started":1}
I20260808 15:55:01.431010 30404 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:54876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:55:01.432111 30404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:01.439309 30543 tablet_service.cc:1467] Processing CreateTablet for tablet 0a953e30393e45659f8cc06dc1d78ed2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcba1451a1544b3cb4ca76de51358af3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:55:01.439709 30682 tablet_service.cc:1467] Processing CreateTablet for tablet 0a953e30393e45659f8cc06dc1d78ed2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcba1451a1544b3cb4ca76de51358af3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:55:01.439728 30826 tablet_service.cc:1467] Processing CreateTablet for tablet 0a953e30393e45659f8cc06dc1d78ed2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dcba1451a1544b3cb4ca76de51358af3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:55:01.440464 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a953e30393e45659f8cc06dc1d78ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:01.440495 30682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a953e30393e45659f8cc06dc1d78ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:01.440563 30826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a953e30393e45659f8cc06dc1d78ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:01.446974 30911 tablet_bootstrap.cc:492] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: Bootstrap starting.
I20260808 15:55:01.448153 30912 tablet_bootstrap.cc:492] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: Bootstrap starting.
I20260808 15:55:01.448431 30911 tablet_bootstrap.cc:654] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:01.449999 30912 tablet_bootstrap.cc:654] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:01.450675 30911 tablet_bootstrap.cc:492] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: No bootstrap required, opened a new log
I20260808 15:55:01.450856 30911 ts_tablet_manager.cc:1397] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 15:55:01.451198 30910 tablet_bootstrap.cc:492] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: Bootstrap starting.
I20260808 15:55:01.451696 30911 raft_consensus.cc:348] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.451921 30911 raft_consensus.cc:374] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:01.452008 30911 raft_consensus.cc:729] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616cba7de7104feb99646f05b2167b14, State: Initialized, Role: FOLLOWER
I20260808 15:55:01.452302 30912 tablet_bootstrap.cc:492] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: No bootstrap required, opened a new log
I20260808 15:55:01.452288 30911 consensus_queue.cc:260] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.452432 30912 ts_tablet_manager.cc:1397] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260808 15:55:01.452901 30911 ts_tablet_manager.cc:1428] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 15:55:01.452935 30910 tablet_bootstrap.cc:654] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:01.452751 30918 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.453123 30918 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.453315 30912 raft_consensus.cc:348] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.453616 30912 raft_consensus.cc:374] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:01.453776 30912 raft_consensus.cc:729] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Initialized, Role: FOLLOWER
I20260808 15:55:01.454066 30912 consensus_queue.cc:260] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.454510 30916 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.454825 30916 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.455425 30910 tablet_bootstrap.cc:492] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: No bootstrap required, opened a new log
I20260808 15:55:01.455564 30910 ts_tablet_manager.cc:1397] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:55:01.455839 30912 ts_tablet_manager.cc:1428] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:01.456640 30910 raft_consensus.cc:348] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.456918 30910 raft_consensus.cc:374] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:01.457023 30910 raft_consensus.cc:729] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87e996db5b48b9b545fc1a379e26f3, State: Initialized, Role: FOLLOWER
I20260808 15:55:01.457281 30910 consensus_queue.cc:260] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.457794 30917 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.458199 30917 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.458684 30910 ts_tablet_manager.cc:1428] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:01.723847 30916 raft_consensus.cc:482] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:01.724058 30916 raft_consensus.cc:504] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.724651 30916 leader_election.cc:288] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:01.725342 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a953e30393e45659f8cc06dc1d78ed2" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14" is_pre_election: true
I20260808 15:55:01.725344 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a953e30393e45659f8cc06dc1d78ed2" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3" is_pre_election: true
I20260808 15:55:01.725628 30706 raft_consensus.cc:2454] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:01.725625 30846 raft_consensus.cc:2454] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:01.726125 30494 leader_election.cc:302] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 616cba7de7104feb99646f05b2167b14, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:01.726454 30916 raft_consensus.cc:2781] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:01.726575 30916 raft_consensus.cc:482] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:01.726708 30916 raft_consensus.cc:3037] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:01.728731 30916 raft_consensus.cc:504] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.729332 30916 leader_election.cc:288] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:01.729822 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a953e30393e45659f8cc06dc1d78ed2" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14"
I20260808 15:55:01.729862 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a953e30393e45659f8cc06dc1d78ed2" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3"
I20260808 15:55:01.730036 30846 raft_consensus.cc:3037] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:01.730145 30706 raft_consensus.cc:3037] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:01.731997 30846 raft_consensus.cc:2454] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:01.732159 30706 raft_consensus.cc:2454] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:01.732434 30495 leader_election.cc:302] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616cba7de7104feb99646f05b2167b14, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:01.732774 30916 raft_consensus.cc:2781] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:01.732963 30916 raft_consensus.cc:686] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Running, Role: LEADER
I20260808 15:55:01.733225 30916 consensus_queue.cc:237] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:01.734771 30921 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: TxnStatusTablet state changed. Reason: New leader 7bef63b80d9449068e40d8da73d4afc9. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.735090 30921 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:55:01.735852 30933 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:55:01.736166 30933 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:55:01.737177 30406 catalog_manager.cc:5495] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1). New cstate: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:01.827437 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.820025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37653 (local address 127.0.0.1:58080)
0808 15:55:01.820248 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.820270 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.820409 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.821261 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.821267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.821281 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:55:01.821531 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.821543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.822865 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.822871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.824520 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.824537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.825324 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.825331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.825587 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.825677 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:55:01.826873 (+ 1196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:55:01.826896 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.826982 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.827141 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 15:55:01.827490 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.820250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.27.254.1:37653)
0808 15:55:01.820845 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.820850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.820872 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:55:01.820961 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.820965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.821047 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.821214 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:01.821704 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.822726 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.824685 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.825160 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.825278 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.825927 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:01.826760 (+ 833us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:01.826915 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.827004 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.827084 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":353,"threads_started":1}
I20260808 15:55:01.828429 30921 consensus_queue.cc:1035] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:01.837517 30846 raft_consensus.cc:1261] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Refusing update from remote peer 7bef63b80d9449068e40d8da73d4afc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:01.838762 30940 consensus_queue.cc:1035] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:01.840830 30916 consensus_queue.cc:1035] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:01.842928 30705 raft_consensus.cc:1261] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Refusing update from remote peer 7bef63b80d9449068e40d8da73d4afc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:01.843859 30916 consensus_queue.cc:1035] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:01.874782 30917 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: TxnStatusTablet state changed. Reason: New leader 7bef63b80d9449068e40d8da73d4afc9. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.875092 30917 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.879733 30918 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: TxnStatusTablet state changed. Reason: New leader 7bef63b80d9449068e40d8da73d4afc9. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.880002 30918 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.880406 30938 mvcc.cc:205] Tried to move back new op lower bound from 7316293639514284032 to 7316293639101476864. Current Snapshot: MvccSnapshot[applied={T|T < 7316293639514284032}]
I20260808 15:55:01.885324 30941 mvcc.cc:205] Tried to move back new op lower bound from 7316293639514284032 to 7316293639101476864. Current Snapshot: MvccSnapshot[applied={T|T < 7316293639514284032}]
I20260808 15:55:01.886118 30943 mvcc.cc:205] Tried to move back new op lower bound from 7316293639514284032 to 7316293639101476864. Current Snapshot: MvccSnapshot[applied={T|T < 7316293639514284032}]
I20260808 15:55:01.893085 30921 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.893385 30921 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:55:01.893815 30918 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.894063 30918 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 616cba7de7104feb99646f05b2167b14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.896777 30917 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.897029 30917 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 1a87e996db5b48b9b545fc1a379e26f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:01.897222 30916 tablet_replica.cc:439] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } } }
I20260808 15:55:01.897514 30916 tablet_replica.cc:442] T 0a953e30393e45659f8cc06dc1d78ed2 P 7bef63b80d9449068e40d8da73d4afc9: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:55:01.911096 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.905105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.27.254.1:37653)
0808 15:55:01.905264 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.905268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.905792 (+ 524us) server_negotiation.cc:407] Connection header received
0808 15:55:01.906000 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.906006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.906087 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.906190 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:01.906599 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.907322 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.909239 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.909796 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.909961 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.910368 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:01.910468 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:01.910570 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.910713 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.910809 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:55:01.927971 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.922208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:47509 (local address 127.27.254.1:37653)
0808 15:55:01.922440 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:01.922449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:01.922967 (+ 518us) server_negotiation.cc:407] Connection header received
0808 15:55:01.923169 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:01.923174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:01.923269 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:01.923364 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:01.923801 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.924570 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.926152 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.926601 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.926739 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.927190 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:01.927290 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:01.927384 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:01.927513 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 15:55:01.927590 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260808 15:55:01.928025 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:01.921981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37653 (local address 127.27.254.1:47509)
0808 15:55:01.922870 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:01.922885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:01.923049 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:01.923380 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:01.923384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:01.923394 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:55:01.923654 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.923663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.924710 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.924715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.925970 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:01.925983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:01.926791 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:01.926799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:01.926933 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:01.926989 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:55:01.927384 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:55:01.927396 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 15:55:01.927495 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 15:55:01.927631 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":597,"threads_started":1}
I20260808 15:55:02.004339 30404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:02.004660 30404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:02.008495 30404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Sending DeleteTablet for 3 replicas of tablet db5f4917de5d4327a22f3c6fe7574402
I20260808 15:55:02.009747 30682 tablet_service.cc:1514] Processing DeleteTablet for tablet db5f4917de5d4327a22f3c6fe7574402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:53072
I20260808 15:55:02.009951 30826 tablet_service.cc:1514] Processing DeleteTablet for tablet db5f4917de5d4327a22f3c6fe7574402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33352
I20260808 15:55:02.010416 30543 tablet_service.cc:1514] Processing DeleteTablet for tablet db5f4917de5d4327a22f3c6fe7574402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:58076
I20260808 15:55:02.015816 30404 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:02.017349 30404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:02.021999 30959 tablet_replica.cc:330] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: stopping tablet replica
I20260808 15:55:02.022214 30958 tablet_replica.cc:330] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: stopping tablet replica
I20260808 15:55:02.022507 30959 raft_consensus.cc:2229] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:02.022583 30958 raft_consensus.cc:2229] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:02.022939 30958 raft_consensus.cc:2258] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:02.025022 30958 ts_tablet_manager.cc:1905] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:02.026372 30957 tablet_replica.cc:330] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: stopping tablet replica
I20260808 15:55:02.026949 30957 raft_consensus.cc:2229] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:02.027352 30957 raft_consensus.cc:2258] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:02.027356 30825 tablet_service.cc:1467] Processing CreateTablet for tablet 16a36cea3e804217bdebee2dabe33e20 (DEFAULT_TABLE table=test-table [id=7698562e0a9245b6a4450458577729d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:02.027640 30543 tablet_service.cc:1467] Processing CreateTablet for tablet 16a36cea3e804217bdebee2dabe33e20 (DEFAULT_TABLE table=test-table [id=7698562e0a9245b6a4450458577729d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:02.027832 30825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a36cea3e804217bdebee2dabe33e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:02.028100 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a36cea3e804217bdebee2dabe33e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:02.029511 30957 ts_tablet_manager.cc:1905] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:02.034155 30682 tablet_service.cc:1467] Processing CreateTablet for tablet 16a36cea3e804217bdebee2dabe33e20 (DEFAULT_TABLE table=test-table [id=7698562e0a9245b6a4450458577729d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:02.034618 30682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a36cea3e804217bdebee2dabe33e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:02.037719 30957 ts_tablet_manager.cc:1918] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:02.037940 30957 log.cc:1196] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/db5f4917de5d4327a22f3c6fe7574402
I20260808 15:55:02.038632 30957 ts_tablet_manager.cc:1939] T db5f4917de5d4327a22f3c6fe7574402 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting consensus metadata
I20260808 15:55:02.040689 30391 catalog_manager.cc:4841] TS 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511): tablet db5f4917de5d4327a22f3c6fe7574402 (table test-table [id=1607894c86bd4a4ab222c22d02fd4966]) successfully deleted
I20260808 15:55:02.051196 30960 tablet_bootstrap.cc:492] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Bootstrap starting.
I20260808 15:55:02.053269 30960 tablet_bootstrap.cc:654] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:02.053550 30958 ts_tablet_manager.cc:1918] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:02.053493 30959 raft_consensus.cc:2258] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:02.053762 30958 log.cc:1196] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/db5f4917de5d4327a22f3c6fe7574402
I20260808 15:55:02.054863 30961 tablet_bootstrap.cc:492] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Bootstrap starting.
I20260808 15:55:02.056057 30959 ts_tablet_manager.cc:1905] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:02.056727 30961 tablet_bootstrap.cc:654] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:02.062681 30960 tablet_bootstrap.cc:492] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: No bootstrap required, opened a new log
I20260808 15:55:02.062916 30960 ts_tablet_manager.cc:1397] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20260808 15:55:02.063777 30960 raft_consensus.cc:348] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.064152 30960 raft_consensus.cc:374] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:02.064316 30960 raft_consensus.cc:729] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616cba7de7104feb99646f05b2167b14, State: Initialized, Role: FOLLOWER
I20260808 15:55:02.062716 30958 ts_tablet_manager.cc:1939] T db5f4917de5d4327a22f3c6fe7574402 P 616cba7de7104feb99646f05b2167b14: Deleting consensus metadata
I20260808 15:55:02.064524 30959 ts_tablet_manager.cc:1918] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:02.064742 30959 log.cc:1196] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/db5f4917de5d4327a22f3c6fe7574402
I20260808 15:55:02.065342 30959 ts_tablet_manager.cc:1939] T db5f4917de5d4327a22f3c6fe7574402 P 7bef63b80d9449068e40d8da73d4afc9: Deleting consensus metadata
I20260808 15:55:02.066640 30392 catalog_manager.cc:4841] TS 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107): tablet db5f4917de5d4327a22f3c6fe7574402 (table test-table [id=1607894c86bd4a4ab222c22d02fd4966]) successfully deleted
I20260808 15:55:02.066920 30391 catalog_manager.cc:4841] TS 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653): tablet db5f4917de5d4327a22f3c6fe7574402 (table test-table [id=1607894c86bd4a4ab222c22d02fd4966]) successfully deleted
I20260808 15:55:02.067200 30960 consensus_queue.cc:260] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.069294 30960 ts_tablet_manager.cc:1428] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260808 15:55:02.078889 30963 tablet_bootstrap.cc:492] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Bootstrap starting.
I20260808 15:55:02.079744 30961 tablet_bootstrap.cc:492] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: No bootstrap required, opened a new log
I20260808 15:55:02.079926 30961 ts_tablet_manager.cc:1397] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.012s
I20260808 15:55:02.080967 30963 tablet_bootstrap.cc:654] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:02.080897 30961 raft_consensus.cc:348] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.081180 30961 raft_consensus.cc:374] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:02.081276 30961 raft_consensus.cc:729] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87e996db5b48b9b545fc1a379e26f3, State: Initialized, Role: FOLLOWER
I20260808 15:55:02.081566 30961 consensus_queue.cc:260] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.082197 30961 ts_tablet_manager.cc:1428] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 15:55:02.089702 30963 tablet_bootstrap.cc:492] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: No bootstrap required, opened a new log
I20260808 15:55:02.089896 30963 ts_tablet_manager.cc:1397] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260808 15:55:02.090629 30963 raft_consensus.cc:348] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.090884 30963 raft_consensus.cc:374] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:02.090996 30963 raft_consensus.cc:729] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Initialized, Role: FOLLOWER
I20260808 15:55:02.091285 30963 consensus_queue.cc:260] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.091912 30963 ts_tablet_manager.cc:1428] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 15:55:02.245117 30921 raft_consensus.cc:482] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:02.245390 30921 raft_consensus.cc:504] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.246169 30921 leader_election.cc:288] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:02.246794 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a36cea3e804217bdebee2dabe33e20" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3" is_pre_election: true
I20260808 15:55:02.246826 30845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a36cea3e804217bdebee2dabe33e20" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14" is_pre_election: true
I20260808 15:55:02.247093 30845 raft_consensus.cc:2454] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:02.247102 30705 raft_consensus.cc:2454] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:02.247625 30495 leader_election.cc:302] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 616cba7de7104feb99646f05b2167b14, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:02.247967 30921 raft_consensus.cc:2781] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:02.248087 30921 raft_consensus.cc:482] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:02.248227 30921 raft_consensus.cc:3037] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:02.250530 30921 raft_consensus.cc:504] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.251147 30921 leader_election.cc:288] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:02.251559 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a36cea3e804217bdebee2dabe33e20" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3"
I20260808 15:55:02.251801 30705 raft_consensus.cc:3037] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:02.251745 30845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a36cea3e804217bdebee2dabe33e20" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14"
I20260808 15:55:02.251953 30845 raft_consensus.cc:3037] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:02.254942 30845 raft_consensus.cc:2454] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:02.254930 30705 raft_consensus.cc:2454] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:02.255374 30494 leader_election.cc:302] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:02.255649 30921 raft_consensus.cc:2781] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:02.255829 30921 raft_consensus.cc:686] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Running, Role: LEADER
I20260808 15:55:02.256074 30921 consensus_queue.cc:237] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:02.259007 30404 catalog_manager.cc:5495] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1). New cstate: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:02.296063 30706 raft_consensus.cc:1261] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Refusing update from remote peer 7bef63b80d9449068e40d8da73d4afc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:02.296173 30845 raft_consensus.cc:1261] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Refusing update from remote peer 7bef63b80d9449068e40d8da73d4afc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:02.296659 30940 consensus_queue.cc:1035] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:02.296941 30916 consensus_queue.cc:1035] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:55:02.319515 30938 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 15:55:03.913442 30405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:03.913654 30405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:03.917634 30405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Sending DeleteTablet for 3 replicas of tablet 16a36cea3e804217bdebee2dabe33e20
I20260808 15:55:03.918339 30543 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a36cea3e804217bdebee2dabe33e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:03 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:58076
I20260808 15:55:03.918448 30825 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a36cea3e804217bdebee2dabe33e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:03 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33352
I20260808 15:55:03.918484 30682 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a36cea3e804217bdebee2dabe33e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:03 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:53072
I20260808 15:55:03.919082 30990 tablet_replica.cc:330] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: stopping tablet replica
I20260808 15:55:03.919287 30990 raft_consensus.cc:2229] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:03.919528 30990 raft_consensus.cc:2258] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:03.920339 30990 ts_tablet_manager.cc:1905] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:03.926584 30990 ts_tablet_manager.cc:1918] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:55:03.926723 30990 log.cc:1196] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/16a36cea3e804217bdebee2dabe33e20
I20260808 15:55:03.927214 30990 ts_tablet_manager.cc:1939] T 16a36cea3e804217bdebee2dabe33e20 P 616cba7de7104feb99646f05b2167b14: Deleting consensus metadata
I20260808 15:55:03.928340 30392 catalog_manager.cc:4841] TS 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107): tablet 16a36cea3e804217bdebee2dabe33e20 (table test-table [id=7698562e0a9245b6a4450458577729d0]) successfully deleted
I20260808 15:55:03.929986 30989 tablet_replica.cc:330] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: stopping tablet replica
I20260808 15:55:03.930181 30991 tablet_replica.cc:330] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: stopping tablet replica
I20260808 15:55:03.930349 30989 raft_consensus.cc:2229] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:03.930373 30991 raft_consensus.cc:2229] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:03.930590 30991 raft_consensus.cc:2258] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:03.930891 30989 raft_consensus.cc:2258] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:03.931931 30989 ts_tablet_manager.cc:1905] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:03.933019 30991 ts_tablet_manager.cc:1905] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:03.938925 30989 ts_tablet_manager.cc:1918] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:55:03.939062 30989 log.cc:1196] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/16a36cea3e804217bdebee2dabe33e20
I20260808 15:55:03.939563 30989 ts_tablet_manager.cc:1939] T 16a36cea3e804217bdebee2dabe33e20 P 7bef63b80d9449068e40d8da73d4afc9: Deleting consensus metadata
I20260808 15:55:03.939670 30991 ts_tablet_manager.cc:1918] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:55:03.939814 30991 log.cc:1196] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/16a36cea3e804217bdebee2dabe33e20
I20260808 15:55:03.940245 30991 ts_tablet_manager.cc:1939] T 16a36cea3e804217bdebee2dabe33e20 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting consensus metadata
I20260808 15:55:03.940876 30391 catalog_manager.cc:4841] TS 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653): tablet 16a36cea3e804217bdebee2dabe33e20 (table test-table [id=7698562e0a9245b6a4450458577729d0]) successfully deleted
I20260808 15:55:03.941350 30391 catalog_manager.cc:4841] TS 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511): tablet 16a36cea3e804217bdebee2dabe33e20 (table test-table [id=7698562e0a9245b6a4450458577729d0]) successfully deleted
Aug 08 15:55:03 dist-test-slave-srgv krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:55:03 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:03.982256 30999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:03.959742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.27.254.62:37771)
0808 15:55:03.960284 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:03.960288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:03.960342 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:55:03.960523 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:03.960527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:03.960600 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:03.960709 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:03.962105 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:03.962890 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:03.964101 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:03.964485 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:03.964581 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:03.976940 (+ 12359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:03.976946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:03.976950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:03.976994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:03.979707 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:03.980486 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:03.980490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:03.980492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:03.980623 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:03.980980 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:03.980983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:03.980985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:03.981256 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:03.981517 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:03.981821 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 15:55:03.981991 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":290,"threads_started":1}
I20260808 15:55:03.986428 30405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:03.987160 30405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:03.993856 30543 tablet_service.cc:1467] Processing CreateTablet for tablet a4851a24896c426d998a1acbf5a28ae1 (DEFAULT_TABLE table=test-table [id=a751e93e7ac34f6e90e3886e852cbaec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:03.994128 30825 tablet_service.cc:1467] Processing CreateTablet for tablet a4851a24896c426d998a1acbf5a28ae1 (DEFAULT_TABLE table=test-table [id=a751e93e7ac34f6e90e3886e852cbaec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:03.994226 30682 tablet_service.cc:1467] Processing CreateTablet for tablet a4851a24896c426d998a1acbf5a28ae1 (DEFAULT_TABLE table=test-table [id=a751e93e7ac34f6e90e3886e852cbaec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:03.994396 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4851a24896c426d998a1acbf5a28ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:03.994599 30825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4851a24896c426d998a1acbf5a28ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:03.994654 30682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4851a24896c426d998a1acbf5a28ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:03.999568 31000 tablet_bootstrap.cc:492] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Bootstrap starting.
I20260808 15:55:04.001552 31002 tablet_bootstrap.cc:492] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Bootstrap starting.
I20260808 15:55:04.002206 31000 tablet_bootstrap.cc:654] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:04.003455 31002 tablet_bootstrap.cc:654] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:04.005921 31000 tablet_bootstrap.cc:492] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: No bootstrap required, opened a new log
I20260808 15:55:04.005921 31002 tablet_bootstrap.cc:492] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: No bootstrap required, opened a new log
I20260808 15:55:04.005936 31001 tablet_bootstrap.cc:492] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Bootstrap starting.
I20260808 15:55:04.006072 31000 ts_tablet_manager.cc:1397] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260808 15:55:04.006072 31002 ts_tablet_manager.cc:1397] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 15:55:04.006851 31002 raft_consensus.cc:348] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.006851 31000 raft_consensus.cc:348] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.007099 31002 raft_consensus.cc:374] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:04.007104 31000 raft_consensus.cc:374] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:04.007200 31002 raft_consensus.cc:729] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87e996db5b48b9b545fc1a379e26f3, State: Initialized, Role: FOLLOWER
I20260808 15:55:04.007207 31000 raft_consensus.cc:729] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Initialized, Role: FOLLOWER
I20260808 15:55:04.007455 31002 consensus_queue.cc:260] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.007433 31000 consensus_queue.cc:260] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.008116 31000 ts_tablet_manager.cc:1428] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 15:55:04.008716 31001 tablet_bootstrap.cc:654] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:04.008831 31002 ts_tablet_manager.cc:1428] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:04.011036 31001 tablet_bootstrap.cc:492] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: No bootstrap required, opened a new log
I20260808 15:55:04.011201 31001 ts_tablet_manager.cc:1397] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 15:55:04.012032 31001 raft_consensus.cc:348] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.012311 31001 raft_consensus.cc:374] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:04.012415 31001 raft_consensus.cc:729] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616cba7de7104feb99646f05b2167b14, State: Initialized, Role: FOLLOWER
I20260808 15:55:04.012665 31001 consensus_queue.cc:260] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.013748 31001 ts_tablet_manager.cc:1428] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:55:04.188421 31005 raft_consensus.cc:482] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:04.188719 31005 raft_consensus.cc:504] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.190372 31005 leader_election.cc:288] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:04.197364 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.190596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37653 (local address 127.27.254.2:56919)
0808 15:55:04.191277 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:04.191297 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:04.191442 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:04.192343 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:04.192348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:04.192375 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:04.192708 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.192716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.193866 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.193873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.195285 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.195297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.196544 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.196551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.196727 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.196751 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:55:04.196862 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 15:55:04.196997 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":275,"threads_started":1}
I20260808 15:55:04.197355 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.190897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:56919 (local address 127.27.254.1:37653)
0808 15:55:04.192020 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:04.192025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:04.192045 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:55:04.192128 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:04.192131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:04.192190 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:04.192325 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:04.192887 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.193718 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.195453 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.196334 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.196539 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.196616 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:04.196889 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 15:55:04.197017 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":959,"threads_started":1}
I20260808 15:55:04.197536 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.190603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:38107 (local address 127.27.254.2:44747)
0808 15:55:04.191985 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:04.192007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:04.192166 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:04.192488 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:04.192492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:04.192517 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:04.192940 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.192948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.194069 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.194075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.195472 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.195483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.196653 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.196660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.196822 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.196845 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:55:04.196949 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:55:04.197069 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":513,"thread_start_us":567,"threads_started":1}
I20260808 15:55:04.198063 31009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.190857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44747 (local address 127.27.254.3:38107)
0808 15:55:04.191565 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:04.191573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:04.192104 (+ 531us) server_negotiation.cc:407] Connection header received
0808 15:55:04.192304 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:04.192308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:04.192363 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:04.192464 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:04.193102 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.193916 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.195614 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.196416 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.196623 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.196707 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:04.196988 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 15:55:04.197117 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":354,"threads_started":1}
I20260808 15:55:04.198323 30568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4851a24896c426d998a1acbf5a28ae1" candidate_uuid: "1a87e996db5b48b9b545fc1a379e26f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bef63b80d9449068e40d8da73d4afc9" is_pre_election: true
I20260808 15:55:04.198609 30845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4851a24896c426d998a1acbf5a28ae1" candidate_uuid: "1a87e996db5b48b9b545fc1a379e26f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14" is_pre_election: true
I20260808 15:55:04.198879 30845 raft_consensus.cc:2454] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a87e996db5b48b9b545fc1a379e26f3 in term 0.
I20260808 15:55:04.198858 30568 raft_consensus.cc:2454] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a87e996db5b48b9b545fc1a379e26f3 in term 0.
I20260808 15:55:04.199379 30635 leader_election.cc:302] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 616cba7de7104feb99646f05b2167b14; no voters:
I20260808 15:55:04.199772 31005 raft_consensus.cc:2781] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:04.199900 31005 raft_consensus.cc:482] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:04.200001 31005 raft_consensus.cc:3037] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:04.202224 31005 raft_consensus.cc:504] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.202791 31005 leader_election.cc:288] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [CANDIDATE]: Term 1 election: Requested vote from peers 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:04.203214 30845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4851a24896c426d998a1acbf5a28ae1" candidate_uuid: "1a87e996db5b48b9b545fc1a379e26f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14"
I20260808 15:55:04.203312 30568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4851a24896c426d998a1acbf5a28ae1" candidate_uuid: "1a87e996db5b48b9b545fc1a379e26f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bef63b80d9449068e40d8da73d4afc9"
I20260808 15:55:04.203409 30845 raft_consensus.cc:3037] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:04.203543 30568 raft_consensus.cc:3037] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:04.205364 30845 raft_consensus.cc:2454] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a87e996db5b48b9b545fc1a379e26f3 in term 1.
I20260808 15:55:04.205690 30568 raft_consensus.cc:2454] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a87e996db5b48b9b545fc1a379e26f3 in term 1.
I20260808 15:55:04.205804 30635 leader_election.cc:302] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 616cba7de7104feb99646f05b2167b14; no voters:
I20260808 15:55:04.206099 31005 raft_consensus.cc:2781] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:04.206526 31005 raft_consensus.cc:686] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a87e996db5b48b9b545fc1a379e26f3, State: Running, Role: LEADER
I20260808 15:55:04.206948 31005 consensus_queue.cc:237] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.210232 30404 catalog_manager.cc:5495] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "1a87e996db5b48b9b545fc1a379e26f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:04.256455 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.250452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.27.254.2:40511)
0808 15:55:04.250993 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:04.250997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:04.251014 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:04.251091 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:04.251094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:04.251179 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:04.251358 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:04.251817 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.252621 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.254181 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.254576 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.254720 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.255227 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:04.255804 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:04.255961 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:04.256041 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 15:55:04.256117 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":270,"threads_started":1}
I20260808 15:55:04.271461 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.265956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:58287 (local address 127.27.254.1:37653)
0808 15:55:04.266191 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:04.266199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:04.266501 (+ 302us) server_negotiation.cc:407] Connection header received
0808 15:55:04.266719 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:04.266724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:04.266790 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:04.266891 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:04.267393 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.268223 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.269748 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.270202 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.270353 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.270748 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:04.270846 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:04.270945 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:04.271077 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 15:55:04.271155 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:55:04.271570 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.265782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37653 (local address 127.27.254.2:58287)
0808 15:55:04.266402 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:04.266417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:04.266575 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:04.266909 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:04.266914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:04.266925 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:55:04.267248 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.267256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.268356 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.268364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.269555 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.269567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.270384 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.270391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.270522 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.270572 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:55:04.270945 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:55:04.270959 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 15:55:04.271055 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 15:55:04.271196 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":269,"threads_started":1}
I20260808 15:55:04.279717 30845 raft_consensus.cc:1261] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Refusing update from remote peer 1a87e996db5b48b9b545fc1a379e26f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:04.280009 30568 raft_consensus.cc:1261] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Refusing update from remote peer 1a87e996db5b48b9b545fc1a379e26f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:04.280359 31012 consensus_queue.cc:1035] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:04.280730 31005 consensus_queue.cc:1035] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:04.310891 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.302343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:37805 (local address 127.27.254.2:40511)
0808 15:55:04.302527 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:04.302533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:04.306021 (+ 3488us) server_negotiation.cc:407] Connection header received
0808 15:55:04.306224 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:04.306228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:04.306291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:04.306365 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:04.306813 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.307524 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.309270 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.309864 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.309961 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.310336 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:04.310428 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:04.310494 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:04.310615 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 15:55:04.310677 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:55:04.310956 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.302163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40511 (local address 127.27.254.1:37805)
0808 15:55:04.305929 (+ 3766us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:04.305950 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:04.306085 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:04.306399 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:04.306402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:04.306412 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:55:04.306679 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.306687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.307653 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.307657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.309107 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:04.309118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.310018 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.310023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:04.310150 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.310188 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:55:04.310511 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:55:04.310519 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:55:04.310576 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 15:55:04.310679 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"thread_start_us":3410,"threads_started":1}
I20260808 15:55:04.338608 30404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:04.338873 30404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:04.342020 30404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Sending DeleteTablet for 3 replicas of tablet a4851a24896c426d998a1acbf5a28ae1
I20260808 15:55:04.342860 30682 tablet_service.cc:1514] Processing DeleteTablet for tablet a4851a24896c426d998a1acbf5a28ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:04 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:53072
I20260808 15:55:04.343017 30825 tablet_service.cc:1514] Processing DeleteTablet for tablet a4851a24896c426d998a1acbf5a28ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:04 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33352
I20260808 15:55:04.343044 30543 tablet_service.cc:1514] Processing DeleteTablet for tablet a4851a24896c426d998a1acbf5a28ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:04 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:58076
I20260808 15:55:04.343300 30990 tablet_replica.cc:330] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: stopping tablet replica
I20260808 15:55:04.343319 30991 tablet_replica.cc:330] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: stopping tablet replica
I20260808 15:55:04.343312 30989 tablet_replica.cc:330] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: stopping tablet replica
I20260808 15:55:04.343497 30990 raft_consensus.cc:2229] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:04.343501 30989 raft_consensus.cc:2229] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:04.343509 30991 raft_consensus.cc:2229] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:04.343698 30990 raft_consensus.cc:2258] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:04.343695 30989 raft_consensus.cc:2258] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:04.344030 30991 raft_consensus.cc:2258] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:04.344852 30990 ts_tablet_manager.cc:1905] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:04.345196 30991 ts_tablet_manager.cc:1905] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:04.346524 30989 ts_tablet_manager.cc:1905] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:04.351544 30990 ts_tablet_manager.cc:1918] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:04.351701 30990 log.cc:1196] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/a4851a24896c426d998a1acbf5a28ae1
I20260808 15:55:04.352138 30990 ts_tablet_manager.cc:1939] T a4851a24896c426d998a1acbf5a28ae1 P 616cba7de7104feb99646f05b2167b14: Deleting consensus metadata
I20260808 15:55:04.352006 30404 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:04.352733 30404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:04.353250 30392 catalog_manager.cc:4841] TS 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107): tablet a4851a24896c426d998a1acbf5a28ae1 (table test-table [id=a751e93e7ac34f6e90e3886e852cbaec]) successfully deleted
I20260808 15:55:04.353870 30989 ts_tablet_manager.cc:1918] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:04.354025 30989 log.cc:1196] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/a4851a24896c426d998a1acbf5a28ae1
I20260808 15:55:04.354554 30991 ts_tablet_manager.cc:1918] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:04.354583 30989 ts_tablet_manager.cc:1939] T a4851a24896c426d998a1acbf5a28ae1 P 7bef63b80d9449068e40d8da73d4afc9: Deleting consensus metadata
I20260808 15:55:04.354725 30991 log.cc:1196] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/a4851a24896c426d998a1acbf5a28ae1
I20260808 15:55:04.355136 30991 ts_tablet_manager.cc:1939] T a4851a24896c426d998a1acbf5a28ae1 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting consensus metadata
I20260808 15:55:04.356025 30391 catalog_manager.cc:4841] TS 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653): tablet a4851a24896c426d998a1acbf5a28ae1 (table test-table [id=a751e93e7ac34f6e90e3886e852cbaec]) successfully deleted
I20260808 15:55:04.357733 30391 catalog_manager.cc:4841] TS 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511): tablet a4851a24896c426d998a1acbf5a28ae1 (table test-table [id=a751e93e7ac34f6e90e3886e852cbaec]) successfully deleted
I20260808 15:55:04.360157 30543 tablet_service.cc:1467] Processing CreateTablet for tablet 78c1198b673145e58c8d84b9b99382a4 (DEFAULT_TABLE table=test-table [id=30d24fe5a52c47a3b4b363a6a9a70a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:04.360448 30682 tablet_service.cc:1467] Processing CreateTablet for tablet 78c1198b673145e58c8d84b9b99382a4 (DEFAULT_TABLE table=test-table [id=30d24fe5a52c47a3b4b363a6a9a70a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:04.360452 30825 tablet_service.cc:1467] Processing CreateTablet for tablet 78c1198b673145e58c8d84b9b99382a4 (DEFAULT_TABLE table=test-table [id=30d24fe5a52c47a3b4b363a6a9a70a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:04.360589 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c1198b673145e58c8d84b9b99382a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:04.360845 30825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c1198b673145e58c8d84b9b99382a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:04.360846 30682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c1198b673145e58c8d84b9b99382a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:04.367007 31000 tablet_bootstrap.cc:492] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Bootstrap starting.
I20260808 15:55:04.367590 31001 tablet_bootstrap.cc:492] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Bootstrap starting.
I20260808 15:55:04.368716 31000 tablet_bootstrap.cc:654] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:04.369240 31001 tablet_bootstrap.cc:654] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:04.370776 31002 tablet_bootstrap.cc:492] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Bootstrap starting.
I20260808 15:55:04.371860 31000 tablet_bootstrap.cc:492] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: No bootstrap required, opened a new log
I20260808 15:55:04.372027 31000 ts_tablet_manager.cc:1397] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260808 15:55:04.372561 31002 tablet_bootstrap.cc:654] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:04.372891 31000 raft_consensus.cc:348] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.373164 31000 raft_consensus.cc:374] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:04.373265 31000 raft_consensus.cc:729] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Initialized, Role: FOLLOWER
I20260808 15:55:04.373618 31000 consensus_queue.cc:260] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.375344 31000 ts_tablet_manager.cc:1428] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 15:55:04.382197 31001 tablet_bootstrap.cc:492] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: No bootstrap required, opened a new log
I20260808 15:55:04.382457 31001 ts_tablet_manager.cc:1397] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260808 15:55:04.383311 31001 raft_consensus.cc:348] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.383670 31001 raft_consensus.cc:374] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:04.383805 31001 raft_consensus.cc:729] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616cba7de7104feb99646f05b2167b14, State: Initialized, Role: FOLLOWER
I20260808 15:55:04.384140 31001 consensus_queue.cc:260] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.385313 31002 tablet_bootstrap.cc:492] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: No bootstrap required, opened a new log
I20260808 15:55:04.385591 31002 ts_tablet_manager.cc:1397] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260808 15:55:04.385855 31001 ts_tablet_manager.cc:1428] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 15:55:04.386498 31002 raft_consensus.cc:348] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.386792 31002 raft_consensus.cc:374] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:04.386883 31002 raft_consensus.cc:729] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87e996db5b48b9b545fc1a379e26f3, State: Initialized, Role: FOLLOWER
I20260808 15:55:04.387156 31002 consensus_queue.cc:260] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.387784 31002 ts_tablet_manager.cc:1428] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 15:55:04.398794 30978 raft_consensus.cc:482] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:04.398998 30978 raft_consensus.cc:504] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.399758 30978 leader_election.cc:288] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:04.400236 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c1198b673145e58c8d84b9b99382a4" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3" is_pre_election: true
I20260808 15:55:04.400498 30705 raft_consensus.cc:2454] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:04.400465 30845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c1198b673145e58c8d84b9b99382a4" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14" is_pre_election: true
I20260808 15:55:04.400717 30845 raft_consensus.cc:2454] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 0.
I20260808 15:55:04.401057 30494 leader_election.cc:302] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a87e996db5b48b9b545fc1a379e26f3, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:04.401341 30978 raft_consensus.cc:2781] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:04.401482 30978 raft_consensus.cc:482] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:04.401566 30978 raft_consensus.cc:3037] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:04.404510 30978 raft_consensus.cc:504] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.405117 30978 leader_election.cc:288] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511), 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107)
I20260808 15:55:04.405656 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c1198b673145e58c8d84b9b99382a4" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a87e996db5b48b9b545fc1a379e26f3"
I20260808 15:55:04.405694 30845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c1198b673145e58c8d84b9b99382a4" candidate_uuid: "7bef63b80d9449068e40d8da73d4afc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616cba7de7104feb99646f05b2167b14"
I20260808 15:55:04.405882 30705 raft_consensus.cc:3037] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:04.405910 30845 raft_consensus.cc:3037] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:04.408034 30845 raft_consensus.cc:2454] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:04.408563 30495 leader_election.cc:302] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616cba7de7104feb99646f05b2167b14, 7bef63b80d9449068e40d8da73d4afc9; no voters:
I20260808 15:55:04.408878 30978 raft_consensus.cc:2781] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:04.409026 30978 raft_consensus.cc:686] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7bef63b80d9449068e40d8da73d4afc9, State: Running, Role: LEADER
I20260808 15:55:04.409078 30705 raft_consensus.cc:2454] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bef63b80d9449068e40d8da73d4afc9 in term 1.
I20260808 15:55:04.409384 30978 consensus_queue.cc:237] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } }
I20260808 15:55:04.412338 30405 catalog_manager.cc:5495] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1). New cstate: current_term: 1 leader_uuid: "7bef63b80d9449068e40d8da73d4afc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bef63b80d9449068e40d8da73d4afc9" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:04.436826 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:04.431211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.27.254.1:37653)
0808 15:55:04.431432 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:04.431439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:04.431459 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:55:04.431669 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:04.431675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:04.431760 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:04.431850 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:04.432197 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.433363 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.434786 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:04.435493 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:04.435625 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:04.436036 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:04.436190 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:04.436275 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:04.436416 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 15:55:04.436509 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 15:55:04.446264 30845 raft_consensus.cc:1261] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Refusing update from remote peer 7bef63b80d9449068e40d8da73d4afc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:04.446316 30705 raft_consensus.cc:1261] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Refusing update from remote peer 7bef63b80d9449068e40d8da73d4afc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:04.446801 30978 consensus_queue.cc:1035] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616cba7de7104feb99646f05b2167b14" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:04.447206 30921 consensus_queue.cc:1035] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a87e996db5b48b9b545fc1a379e26f3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:55:04.466361 30970 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 15:55:06.045145 30404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:06.045374 30404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:06.049499 30404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b66160b82eec44518558c99eba80a0ce: Sending DeleteTablet for 3 replicas of tablet 78c1198b673145e58c8d84b9b99382a4
I20260808 15:55:06.050230 30543 tablet_service.cc:1514] Processing DeleteTablet for tablet 78c1198b673145e58c8d84b9b99382a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:06 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:58076
I20260808 15:55:06.050577 30682 tablet_service.cc:1514] Processing DeleteTablet for tablet 78c1198b673145e58c8d84b9b99382a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:06 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:53072
I20260808 15:55:06.050705 30825 tablet_service.cc:1514] Processing DeleteTablet for tablet 78c1198b673145e58c8d84b9b99382a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:06 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33352
I20260808 15:55:06.053321 31053 tablet_replica.cc:330] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: stopping tablet replica
I20260808 15:55:06.053611 31053 raft_consensus.cc:2229] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:06.053922 31053 raft_consensus.cc:2258] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:06.054183 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30477
I20260808 15:55:06.054800 31052 tablet_replica.cc:330] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: stopping tablet replica
I20260808 15:55:06.055001 31052 raft_consensus.cc:2229] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:06.055074 31054 tablet_replica.cc:330] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: stopping tablet replica
I20260808 15:55:06.055264 31054 raft_consensus.cc:2229] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:06.055415 31052 raft_consensus.cc:2258] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:06.056102 31054 raft_consensus.cc:2258] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:06.056319 31053 ts_tablet_manager.cc:1905] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:06.056821 31052 ts_tablet_manager.cc:1905] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:06.063930 31052 ts_tablet_manager.cc:1918] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:55:06.064620 31054 ts_tablet_manager.cc:1905] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:06.065537 31052 log.cc:1196] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/78c1198b673145e58c8d84b9b99382a4
I20260808 15:55:06.066058 31052 ts_tablet_manager.cc:1939] T 78c1198b673145e58c8d84b9b99382a4 P 7bef63b80d9449068e40d8da73d4afc9: Deleting consensus metadata
I20260808 15:55:06.067356 30391 catalog_manager.cc:4841] TS 7bef63b80d9449068e40d8da73d4afc9 (127.27.254.1:37653): tablet 78c1198b673145e58c8d84b9b99382a4 (table test-table [id=30d24fe5a52c47a3b4b363a6a9a70a63]) successfully deleted
I20260808 15:55:06.071089 31053 ts_tablet_manager.cc:1918] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:55:06.071192 31054 ts_tablet_manager.cc:1918] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:55:06.071264 31053 log.cc:1196] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/78c1198b673145e58c8d84b9b99382a4
I20260808 15:55:06.071327 31054 log.cc:1196] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/78c1198b673145e58c8d84b9b99382a4
I20260808 15:55:06.071746 31054 ts_tablet_manager.cc:1939] T 78c1198b673145e58c8d84b9b99382a4 P 616cba7de7104feb99646f05b2167b14: Deleting consensus metadata
I20260808 15:55:06.071749 31053 ts_tablet_manager.cc:1939] T 78c1198b673145e58c8d84b9b99382a4 P 1a87e996db5b48b9b545fc1a379e26f3: Deleting consensus metadata
I20260808 15:55:06.072726 30391 catalog_manager.cc:4841] TS 1a87e996db5b48b9b545fc1a379e26f3 (127.27.254.2:40511): tablet 78c1198b673145e58c8d84b9b99382a4 (table test-table [id=30d24fe5a52c47a3b4b363a6a9a70a63]) successfully deleted
I20260808 15:55:06.072867 30392 catalog_manager.cc:4841] TS 616cba7de7104feb99646f05b2167b14 (127.27.254.3:38107): tablet 78c1198b673145e58c8d84b9b99382a4 (table test-table [id=30d24fe5a52c47a3b4b363a6a9a70a63]) successfully deleted
Aug 08 15:55:06 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:06.080554 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:06.055397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.27.254.1:37653)
0808 15:55:06.064973 (+ 9576us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:06.064979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:06.065002 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:55:06.065058 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:06.065061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:06.065113 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:06.065253 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:06.066996 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:06.068167 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:06.069913 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:06.070351 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:06.070483 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:06.074754 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:06.074769 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:06.074784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:06.074844 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:06.077890 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:06.078515 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:06.078524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:06.078528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:06.078660 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:06.079048 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:06.079053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:06.079056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:06.079406 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:06.079690 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:06.080000 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 15:55:06.080214 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9434,"thread_start_us":9527,"threads_started":1}
W20260808 15:55:06.081243 30608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58114
W20260808 15:55:06.081859 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:55:06.082077 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 30477
I20260808 15:55:06.108767 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30617
Aug 08 15:55:06 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:55:06.125005 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:06.109629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.27.254.2:40511)
0808 15:55:06.110156 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:06.110162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:06.110181 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:55:06.110242 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:06.110248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:06.110316 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:06.110432 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:06.111644 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:06.112768 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:06.114169 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:06.114827 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:06.114948 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:06.119083 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:06.119101 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:06.119122 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:06.119179 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:06.122710 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:06.123314 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:06.123321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:06.123323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:06.123449 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:06.123760 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:06.123763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:06.123765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:06.124093 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:06.124348 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:06.124594 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 15:55:06.124755 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
W20260808 15:55:06.125684 30747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53086
W20260808 15:55:06.126322 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:55:06.126484 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 30617
I20260808 15:55:06.148825 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30756
Aug 08 15:55:06 dist-test-slave-srgv krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:55:06.164218 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:06.149672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.27.254.3:38107)
0808 15:55:06.150225 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:06.150229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:06.150246 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:06.150299 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:06.150303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:06.150362 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:06.150484 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:06.151854 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:06.152906 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:06.154276 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:06.154638 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:06.154733 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:06.158780 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:06.158794 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:06.158815 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:06.158849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:06.161772 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:06.162414 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:06.162421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:06.162424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:06.162554 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:06.162874 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:06.162877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:06.162879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:06.163221 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:06.163428 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:06.163687 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 15:55:06.163855 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":279,"threads_started":1}
W20260808 15:55:06.164991 30886 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33368
W20260808 15:55:06.165635 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30756: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:55:06.165818 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 30756
I20260808 15:55:06.184208 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 30374
I20260808 15:55:06.185498 30446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:06.300530 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 30374
2026-08-08T15:55:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8589 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:06 dist-test-slave-srgv krb5kdc[31062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:06 dist-test-slave-srgv krb5kdc[31062](info): set up 2 sockets
Aug 08 15:55:06 dist-test-slave-srgv krb5kdc[31062](info): commencing operation
krb5kdc: starting...
W20260808 15:55:07.379498 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:07 dist-test-slave-srgv krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:07.560370 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35827
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:07.869549 31075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:07.869848 31075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:07.870031 31075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:07.877015 31075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:07.877110 31075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:07.877218 31075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:07.877345 31075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:07.877442 31075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:07.887210 31075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:07.888494 31075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:07.889784 31075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:07.901866 31080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:07.901966 31081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:07.901907 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:07.902245 31075 server_base.cc:1034] running on GCE node
I20260808 15:55:07.903131 31075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:07.904496 31075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:07.905681 31075 hybrid_clock.cc:648] HybridClock initialized: now 1786204507905628 us; error 43 us; skew 500 ppm
I20260808 15:55:07.905994 31075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:07 dist-test-slave-srgv krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:07.909094 31075 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:07.910533 31075 webserver.cc:460] Webserver started at http://127.27.254.62:36693/ using document root <none> and password file <none>
I20260808 15:55:07.911145 31075 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:07.911240 31075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:07.911487 31075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:07.914279 31075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "d655f7aeab1e4e3883dedc21efc3d548"
format_stamp: "Formatted at 2026-08-08 15:55:07 on dist-test-slave-srgv"
server_key: "eb17166bc79e95efa81c51c8bb73a640"
server_key_iv: "d76c16bfdc6c2a60d61c2cdd8f41690c"
server_key_version: "encryptionkey@0"
I20260808 15:55:07.914950 31075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "d655f7aeab1e4e3883dedc21efc3d548"
format_stamp: "Formatted at 2026-08-08 15:55:07 on dist-test-slave-srgv"
server_key: "eb17166bc79e95efa81c51c8bb73a640"
server_key_iv: "d76c16bfdc6c2a60d61c2cdd8f41690c"
server_key_version: "encryptionkey@0"
I20260808 15:55:07.919838 31075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:07.922933 31090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:07.924090 31075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 15:55:07.924214 31075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "d655f7aeab1e4e3883dedc21efc3d548"
format_stamp: "Formatted at 2026-08-08 15:55:07 on dist-test-slave-srgv"
server_key: "eb17166bc79e95efa81c51c8bb73a640"
server_key_iv: "d76c16bfdc6c2a60d61c2cdd8f41690c"
server_key_version: "encryptionkey@0"
I20260808 15:55:07.924352 31075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:07.935359 31075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:07.939468 31075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:07.939777 31075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:07.961897 31075 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:35827
I20260808 15:55:07.961925 31141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:35827 every 8 connection(s)
I20260808 15:55:07.963270 31075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:07.966878 31142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:07.967301 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31075
I20260808 15:55:07.967523 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:07.968116 28664 external_mini_cluster.cc:1428] Setting key c13d3c41edb4bfc582367be291598c6a
I20260808 15:55:07.980594 31142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548: Bootstrap starting.
Aug 08 15:55:07 dist-test-slave-srgv krb5kdc[31062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:07.983701 31142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:07.985036 31142 log.cc:824] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:07.988540 31142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548: No bootstrap required, opened a new log
I20260808 15:55:07.992022 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:07.970006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.27.254.62:35827)
0808 15:55:07.970868 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:07.970883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:07.970932 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:55:07.973268 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:07.973295 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:07.973784 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:07.974202 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:07.975429 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:07.977483 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:07.978716 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:07.979704 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:07.979871 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:07.983699 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:07.983710 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:07.983729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:07.983772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:07.987537 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:07.988184 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:07.988192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:07.988200 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:07.988422 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:07.988816 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:07.988823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:07.988829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:07.989595 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:07.989898 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:07.990238 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 15:55:07.990688 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":325,"threads_started":1}
I20260808 15:55:07.997557 31142 raft_consensus.cc:348] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d655f7aeab1e4e3883dedc21efc3d548" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35827 } }
I20260808 15:55:07.997869 31142 raft_consensus.cc:374] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:07.997948 31142 raft_consensus.cc:729] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d655f7aeab1e4e3883dedc21efc3d548, State: Initialized, Role: FOLLOWER
I20260808 15:55:07.998466 31142 consensus_queue.cc:260] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d655f7aeab1e4e3883dedc21efc3d548" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35827 } }
I20260808 15:55:07.998677 31142 raft_consensus.cc:388] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:07.998766 31142 raft_consensus.cc:482] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:07.998883 31142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:08.001042 31142 raft_consensus.cc:504] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d655f7aeab1e4e3883dedc21efc3d548" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35827 } }
I20260808 15:55:08.001647 31142 leader_election.cc:302] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d655f7aeab1e4e3883dedc21efc3d548; no voters:
I20260808 15:55:08.002223 31142 leader_election.cc:288] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:08.002353 31147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:08.003036 31147 raft_consensus.cc:686] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [term 1 LEADER]: Becoming Leader. State: Replica: d655f7aeab1e4e3883dedc21efc3d548, State: Running, Role: LEADER
I20260808 15:55:08.003506 31147 consensus_queue.cc:237] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d655f7aeab1e4e3883dedc21efc3d548" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35827 } }
I20260808 15:55:08.004189 31142 sys_catalog.cc:564] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:08.006374 31149 sys_catalog.cc:455] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d655f7aeab1e4e3883dedc21efc3d548. Latest consensus state: current_term: 1 leader_uuid: "d655f7aeab1e4e3883dedc21efc3d548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d655f7aeab1e4e3883dedc21efc3d548" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35827 } } }
I20260808 15:55:08.006335 31148 sys_catalog.cc:455] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d655f7aeab1e4e3883dedc21efc3d548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d655f7aeab1e4e3883dedc21efc3d548" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35827 } } }
I20260808 15:55:08.006780 31149 sys_catalog.cc:458] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:08.006780 31148 sys_catalog.cc:458] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:08.008970 31153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:08.014314 31153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:08.023630 31153 catalog_manager.cc:1347] Generated new cluster ID: 4223f7b2ec1247fab6ca301ca6acd862
I20260808 15:55:08.023752 31153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:08.043974 31153 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:08.045300 31153 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:08.057731 31153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d655f7aeab1e4e3883dedc21efc3d548: Generated new TSK 0
I20260808 15:55:08.058562 31153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:55:08.110635 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35827
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:08.432202 31168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:08.432492 31168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:08.432596 31168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:08.439738 31168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:08.439833 31168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:08.440022 31168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:08.449584 31168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:08.450835 31168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:08.452096 31168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:08.464812 31174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:08.464775 31173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:08.465197 31176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:08.465654 31168 server_base.cc:1034] running on GCE node
I20260808 15:55:08.466189 31168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:08.467042 31168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:08.468268 31168 hybrid_clock.cc:648] HybridClock initialized: now 1786204508468210 us; error 56 us; skew 500 ppm
I20260808 15:55:08.468578 31168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:08 dist-test-slave-srgv krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:08.471638 31168 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:55:08.473055 31168 webserver.cc:460] Webserver started at http://127.27.254.1:36355/ using document root <none> and password file <none>
I20260808 15:55:08.473793 31168 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:08.473934 31168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:08.474246 31168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:08.476940 31168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "2f50167d25054e688a925448d635bdd1"
format_stamp: "Formatted at 2026-08-08 15:55:08 on dist-test-slave-srgv"
server_key: "180aef4ca279a25fee2a21981ade844d"
server_key_iv: "249095a57096db6efeb3501aefe75bb1"
server_key_version: "encryptionkey@0"
I20260808 15:55:08.477777 31168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "2f50167d25054e688a925448d635bdd1"
format_stamp: "Formatted at 2026-08-08 15:55:08 on dist-test-slave-srgv"
server_key: "180aef4ca279a25fee2a21981ade844d"
server_key_iv: "249095a57096db6efeb3501aefe75bb1"
server_key_version: "encryptionkey@0"
I20260808 15:55:08.482486 31168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:55:08.485780 31183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:08.486997 31168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:55:08.487175 31168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "2f50167d25054e688a925448d635bdd1"
format_stamp: "Formatted at 2026-08-08 15:55:08 on dist-test-slave-srgv"
server_key: "180aef4ca279a25fee2a21981ade844d"
server_key_iv: "249095a57096db6efeb3501aefe75bb1"
server_key_version: "encryptionkey@0"
I20260808 15:55:08.487380 31168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:08.515439 31168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:08.519410 31168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:08.519717 31168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:08.520663 31168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:08.522058 31168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:08.522171 31168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:08.522324 31168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:08.522408 31168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:08.555549 31168 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:43629
I20260808 15:55:08.555641 31295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:43629 every 8 connection(s)
I20260808 15:55:08.556833 31168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:08.559782 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31168
I20260808 15:55:08.559999 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:08.560767 28664 external_mini_cluster.cc:1428] Setting key 3220c56688538875c4000bb230f4ae67
Aug 08 15:55:08 dist-test-slave-srgv krb5kdc[31062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:08.581087 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:08.559609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:44905 (local address 127.27.254.62:35827)
0808 15:55:08.560235 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:08.560242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:08.561196 (+ 954us) server_negotiation.cc:407] Connection header received
0808 15:55:08.563485 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:08.563489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:08.563548 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:08.563646 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:08.565822 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:08.566523 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:08.568973 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:08.569725 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:08.569861 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:08.573642 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:08.573652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:08.573660 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:08.573706 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:08.577559 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:08.578445 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:08.578453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:08.578460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:08.578628 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:08.579064 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:08.579072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:08.579086 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:08.579485 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:08.579650 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:08.580372 (+ 722us) server_negotiation.cc:299] Negotiation successful
0808 15:55:08.580645 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":376,"threads_started":1}
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:55:08.581881 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:08.560159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35827 (local address 127.27.254.1:44905)
0808 15:55:08.561008 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:08.561057 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:08.563191 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:08.563918 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:08.563932 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:08.564755 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:08.565640 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:08.565669 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:08.566678 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:08.566686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:08.568820 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:08.568833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:08.569928 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:08.569933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:08.570161 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:08.570849 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:08.570872 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:08.573346 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:08.577719 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:08.577726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:08.577740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:08.578296 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:08.578747 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:08.578750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:08.578753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:08.578936 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:08.579609 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:08.579615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:08.579952 (+ 337us) client_negotiation.cc:769] Sending connection context
0808 15:55:08.580188 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 15:55:08.580597 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":304,"threads_started":1}
I20260808 15:55:08.583599 31296 heartbeater.cc:347] Connected to a master server at 127.27.254.62:35827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
I20260808 15:55:08.584007 31296 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:08.584920 31296 heartbeater.cc:511] Master 127.27.254.62:35827 requested a full tablet report, sending...
I20260808 15:55:08.587283 31107 ts_manager.cc:194] Registered new tserver with Master: 2f50167d25054e688a925448d635bdd1 (127.27.254.1:43629)
I20260808 15:55:08.588760 31107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:44905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:08.608081 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35827
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:08.921109 31302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:08.921371 31302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:08.921550 31302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:08.929186 31302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:08.929313 31302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:08.929505 31302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:08.938841 31302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:08.940025 31302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:08.941215 31302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:08.954335 31310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:08.954353 31308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:08.954396 31307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:08.955237 31302 server_base.cc:1034] running on GCE node
I20260808 15:55:08.955838 31302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:08.956764 31302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:08.957980 31302 hybrid_clock.cc:648] HybridClock initialized: now 1786204508957936 us; error 39 us; skew 500 ppm
I20260808 15:55:08.958292 31302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:08 dist-test-slave-srgv krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:08.961288 31302 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:08.962894 31302 webserver.cc:460] Webserver started at http://127.27.254.2:35647/ using document root <none> and password file <none>
I20260808 15:55:08.963593 31302 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:08.963716 31302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:08.964010 31302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:08.966814 31302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "f43b2b50f853418b9d403a1d8f2cd638"
format_stamp: "Formatted at 2026-08-08 15:55:08 on dist-test-slave-srgv"
server_key: "dc85d5b2e96933e36c06f79bd0fbf273"
server_key_iv: "6cea0f87cd20e3a21ab3991f31fad265"
server_key_version: "encryptionkey@0"
I20260808 15:55:08.967624 31302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "f43b2b50f853418b9d403a1d8f2cd638"
format_stamp: "Formatted at 2026-08-08 15:55:08 on dist-test-slave-srgv"
server_key: "dc85d5b2e96933e36c06f79bd0fbf273"
server_key_iv: "6cea0f87cd20e3a21ab3991f31fad265"
server_key_version: "encryptionkey@0"
I20260808 15:55:08.972857 31302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:55:08.976263 31317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:08.977526 31302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:08.977684 31302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "f43b2b50f853418b9d403a1d8f2cd638"
format_stamp: "Formatted at 2026-08-08 15:55:08 on dist-test-slave-srgv"
server_key: "dc85d5b2e96933e36c06f79bd0fbf273"
server_key_iv: "6cea0f87cd20e3a21ab3991f31fad265"
server_key_version: "encryptionkey@0"
I20260808 15:55:08.977882 31302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:08.996142 31302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:08.999833 31302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:09.000101 31302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:09.001024 31302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:09.002297 31302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:09.002373 31302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:09.002537 31302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:09.002586 31302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:09.034679 31302 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:38021
I20260808 15:55:09.034714 31429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:38021 every 8 connection(s)
I20260808 15:55:09.035900 31302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:09.044965 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31302
I20260808 15:55:09.045182 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:09.045670 28664 external_mini_cluster.cc:1428] Setting key f6afff98c34319c9462cddb1fad1d859
Aug 08 15:55:09 dist-test-slave-srgv krb5kdc[31062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:09.054662 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:09.038334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44451 (local address 127.27.254.62:35827)
0808 15:55:09.038554 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:09.038561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:09.039639 (+ 1078us) server_negotiation.cc:407] Connection header received
0808 15:55:09.041208 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:09.041212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:09.041273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:09.041354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:09.042644 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.043365 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.044775 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.045238 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.045402 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:09.048942 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:09.048948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:09.048953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:09.048990 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:09.051653 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:09.052375 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:09.052382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:09.052386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:09.052515 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:09.052930 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:09.052936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:09.052939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:09.053211 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:09.053375 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:09.054066 (+ 691us) server_negotiation.cc:299] Negotiation successful
0808 15:55:09.054312 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:55:09.055573 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:09.038827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35827 (local address 127.27.254.2:44451)
0808 15:55:09.039475 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:09.039514 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:09.040991 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:09.041588 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:09.041595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:09.042024 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:09.042511 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:09.042526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.043487 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.043491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:09.044590 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:09.044599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.045501 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.045509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:09.045724 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:09.046407 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:09.046428 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:09.048746 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:09.051797 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:09.051806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:09.051821 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:09.052247 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:09.052625 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:09.052630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:09.052634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:09.052820 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:09.053335 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:09.053344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:09.053727 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 15:55:09.053991 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 15:55:09.054488 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":259,"threads_started":1}
I20260808 15:55:09.056962 31430 heartbeater.cc:347] Connected to a master server at 127.27.254.62:35827
I20260808 15:55:09.057293 31430 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:09.058209 31430 heartbeater.cc:511] Master 127.27.254.62:35827 requested a full tablet report, sending...
I20260808 15:55:09.059887 31107 ts_manager.cc:194] Registered new tserver with Master: f43b2b50f853418b9d403a1d8f2cd638 (127.27.254.2:38021)
I20260808 15:55:09.061146 31107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:44451
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:09.088557 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35827
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:09.407502 31435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:09.407820 31435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:09.407991 31435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:09.415432 31435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:09.415576 31435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:09.415753 31435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:09.424949 31435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:09.426205 31435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:09.427537 31435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:09.440414 31440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:09.440373 31441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:09.440673 31443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:09.441314 31435 server_base.cc:1034] running on GCE node
I20260808 15:55:09.441893 31435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:09.442682 31435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:09.443877 31435 hybrid_clock.cc:648] HybridClock initialized: now 1786204509443837 us; error 29 us; skew 500 ppm
I20260808 15:55:09.444145 31435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:09 dist-test-slave-srgv krb5kdc[31062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:09.447178 31435 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:09.448530 31435 webserver.cc:460] Webserver started at http://127.27.254.3:45671/ using document root <none> and password file <none>
I20260808 15:55:09.449188 31435 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:09.449316 31435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:09.449652 31435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:09.452418 31435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "13f6958fd51c4eb2bf1464f89f320613"
format_stamp: "Formatted at 2026-08-08 15:55:09 on dist-test-slave-srgv"
server_key: "1cecea1901fe23f904e6f10c51316a80"
server_key_iv: "d528cfa02e2bb191360bbbfe699b9f09"
server_key_version: "encryptionkey@0"
I20260808 15:55:09.453197 31435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "13f6958fd51c4eb2bf1464f89f320613"
format_stamp: "Formatted at 2026-08-08 15:55:09 on dist-test-slave-srgv"
server_key: "1cecea1901fe23f904e6f10c51316a80"
server_key_iv: "d528cfa02e2bb191360bbbfe699b9f09"
server_key_version: "encryptionkey@0"
I20260808 15:55:09.458407 31435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:55:09.461750 31450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:09.463006 31435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:55:09.463168 31435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "13f6958fd51c4eb2bf1464f89f320613"
format_stamp: "Formatted at 2026-08-08 15:55:09 on dist-test-slave-srgv"
server_key: "1cecea1901fe23f904e6f10c51316a80"
server_key_iv: "d528cfa02e2bb191360bbbfe699b9f09"
server_key_version: "encryptionkey@0"
I20260808 15:55:09.463390 31435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:09.486245 31435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:09.489982 31435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:09.490316 31435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:09.491331 31435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:09.492607 31435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:09.492677 31435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:09.492837 31435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:09.492897 31435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:09.525713 31435 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:35573
I20260808 15:55:09.525796 31562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:35573 every 8 connection(s)
I20260808 15:55:09.526980 31435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:09.535053 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31435
I20260808 15:55:09.535252 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:09.535770 28664 external_mini_cluster.cc:1428] Setting key 36c6c0332bd409d32eccdb267b1b40aa
Aug 08 15:55:09 dist-test-slave-srgv krb5kdc[31062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:09.546840 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:09.529626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:32893 (local address 127.27.254.62:35827)
0808 15:55:09.529798 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:09.529802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:09.530885 (+ 1083us) server_negotiation.cc:407] Connection header received
0808 15:55:09.532675 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:09.532680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:09.532776 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:09.532889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:09.534381 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.535446 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.537352 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.537965 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.538059 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:09.541513 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:09.541518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:09.541522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:09.541554 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:09.544086 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:09.544774 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:09.544778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:09.544781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:09.544911 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:09.545295 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:09.545298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:09.545300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:09.545649 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:09.545797 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:09.546414 (+ 617us) server_negotiation.cc:299] Negotiation successful
0808 15:55:09.546584 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:55:09.547753 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:09.530046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35827 (local address 127.27.254.3:32893)
0808 15:55:09.530736 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:09.530772 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:09.532416 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:09.533115 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:09.533123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:09.533678 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:09.534211 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:09.534227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.535600 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.535605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:09.537206 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:09.537218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:09.538192 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:09.538199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:09.538508 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:09.539190 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:09.539211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:09.541271 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:09.544221 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:09.544227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:09.544237 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:09.544619 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:09.545015 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:09.545018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:09.545020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:09.545190 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:09.545794 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:09.545800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:09.546086 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 15:55:09.546302 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 15:55:09.546685 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":257,"threads_started":1}
I20260808 15:55:09.549045 31563 heartbeater.cc:347] Connected to a master server at 127.27.254.62:35827
I20260808 15:55:09.549331 31563 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:09.550052 31563 heartbeater.cc:511] Master 127.27.254.62:35827 requested a full tablet report, sending...
I20260808 15:55:09.551553 31107 ts_manager.cc:194] Registered new tserver with Master: 13f6958fd51c4eb2bf1464f89f320613 (127.27.254.3:35573)
I20260808 15:55:09.552328 31107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:32893
I20260808 15:55:09.562860 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:55:09.591836 31296 heartbeater.cc:503] Master 127.27.254.62:35827 was elected leader, sending a full tablet report...
W20260808 15:55:09.593243 31298 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:09.590492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.27.254.62:35827)
0808 15:55:09.590760 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:09.590768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:09.591143 (+ 375us) server_negotiation.cc:407] Connection header received
0808 15:55:09.591385 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:09.591389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:09.591461 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:09.591564 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:09.592927 (+ 1363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60036: BlockingRecv error: recv got EOF from 127.0.0.1:60036 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:55:09.596827 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31168
W20260808 15:55:09.599985 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31168: Not authorized: Client connection negotiation failed: client connection to 127.27.254.1:43629: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:55:09.600152 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31168
I20260808 15:55:09.615684 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31302
W20260808 15:55:09.618530 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31302: Not authorized: Client connection negotiation failed: client connection to 127.27.254.2:38021: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:55:09.618695 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31302
I20260808 15:55:09.633728 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31435
W20260808 15:55:09.636761 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31435: Not authorized: Client connection negotiation failed: client connection to 127.27.254.3:35573: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:55:09.636886 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31435
I20260808 15:55:09.653023 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31075
I20260808 15:55:09.654142 31137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:09.757156 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31075
2026-08-08T15:55:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3453 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:09 dist-test-slave-srgv krb5kdc[31580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:09 dist-test-slave-srgv krb5kdc[31580](info): set up 2 sockets
Aug 08 15:55:09 dist-test-slave-srgv krb5kdc[31580](info): commencing operation
krb5kdc: starting...
W20260808 15:55:10.831570 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:10 dist-test-slave-srgv krb5kdc[31580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:11.020510 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35887
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:11.336942 31593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:11.337234 31593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:11.337423 31593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:11.344653 31593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:11.344760 31593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:11.344863 31593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:11.344965 31593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:11.345036 31593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:11.354967 31593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:35887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:11.356283 31593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:11.357618 31593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:11.369494 31599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:11.369575 31601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:11.369721 31598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:11.370395 31593 server_base.cc:1034] running on GCE node
I20260808 15:55:11.371162 31593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:11.372539 31593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:11.373777 31593 hybrid_clock.cc:648] HybridClock initialized: now 1786204511373735 us; error 56 us; skew 500 ppm
I20260808 15:55:11.374130 31593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:11 dist-test-slave-srgv krb5kdc[31580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:11.377314 31593 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:11.378747 31593 webserver.cc:460] Webserver started at http://127.27.254.62:43605/ using document root <none> and password file <none>
I20260808 15:55:11.379379 31593 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:11.379477 31593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:11.379745 31593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:11.382723 31593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "cf534183ccda47298c7c3a2db41b08ab"
format_stamp: "Formatted at 2026-08-08 15:55:11 on dist-test-slave-srgv"
server_key: "77f75de0e7afa0c8983c60a0c43e7099"
server_key_iv: "f2ebf665fe23e97c874a892154458482"
server_key_version: "encryptionkey@0"
I20260808 15:55:11.383478 31593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "cf534183ccda47298c7c3a2db41b08ab"
format_stamp: "Formatted at 2026-08-08 15:55:11 on dist-test-slave-srgv"
server_key: "77f75de0e7afa0c8983c60a0c43e7099"
server_key_iv: "f2ebf665fe23e97c874a892154458482"
server_key_version: "encryptionkey@0"
I20260808 15:55:11.388469 31593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:55:11.391799 31608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:11.393100 31593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:11.393239 31593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "cf534183ccda47298c7c3a2db41b08ab"
format_stamp: "Formatted at 2026-08-08 15:55:11 on dist-test-slave-srgv"
server_key: "77f75de0e7afa0c8983c60a0c43e7099"
server_key_iv: "f2ebf665fe23e97c874a892154458482"
server_key_version: "encryptionkey@0"
I20260808 15:55:11.393395 31593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:11.405516 31593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:11.409802 31593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:11.410039 31593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:11.428357 31593 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:35887
I20260808 15:55:11.428387 31659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:35887 every 8 connection(s)
I20260808 15:55:11.429726 31593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:11.433332 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:11.437361 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31593
I20260808 15:55:11.437590 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:11.438390 28664 external_mini_cluster.cc:1428] Setting key 5ddd77cacd858ae2b2164a8aee145ab3
I20260808 15:55:11.444939 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab: Bootstrap starting.
I20260808 15:55:11.448494 31660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:11.449700 31660 log.cc:824] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:11.452999 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab: No bootstrap required, opened a new log
Aug 08 15:55:11 dist-test-slave-srgv krb5kdc[31580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:11.460520 31660 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf534183ccda47298c7c3a2db41b08ab" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35887 } }
I20260808 15:55:11.460888 31660 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:11.461027 31660 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf534183ccda47298c7c3a2db41b08ab, State: Initialized, Role: FOLLOWER
I20260808 15:55:11.461701 31660 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf534183ccda47298c7c3a2db41b08ab" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35887 } }
I20260808 15:55:11.461961 31660 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:11.462100 31660 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:11.462263 31660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:11.462388 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:11.440270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.27.254.62:35887)
0808 15:55:11.441055 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:11.441067 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:11.441105 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:55:11.443425 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:11.443447 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:11.443831 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:11.444174 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:11.445990 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:11.447296 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:11.449099 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:11.449883 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:11.450009 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:11.454715 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:11.454723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:11.454737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:11.454779 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:11.458062 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:11.458754 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:11.458769 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:11.458774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:11.458949 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:11.459359 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:11.459366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:11.459373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:11.459900 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:11.460190 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:11.460645 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 15:55:11.461093 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":383,"threads_started":1}
I20260808 15:55:11.465188 31660 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf534183ccda47298c7c3a2db41b08ab" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35887 } }
I20260808 15:55:11.465842 31660 leader_election.cc:302] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf534183ccda47298c7c3a2db41b08ab; no voters:
I20260808 15:55:11.466498 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:55:11.466543 31660 leader_election.cc:288] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:11.466656 31665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:11.467293 31665 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [term 1 LEADER]: Becoming Leader. State: Replica: cf534183ccda47298c7c3a2db41b08ab, State: Running, Role: LEADER
I20260808 15:55:11.467868 31665 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf534183ccda47298c7c3a2db41b08ab" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35887 } }
I20260808 15:55:11.470419 31667 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf534183ccda47298c7c3a2db41b08ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf534183ccda47298c7c3a2db41b08ab" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35887 } } }
I20260808 15:55:11.470816 31667 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:11.475622 31665 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf534183ccda47298c7c3a2db41b08ab. Latest consensus state: current_term: 1 leader_uuid: "cf534183ccda47298c7c3a2db41b08ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf534183ccda47298c7c3a2db41b08ab" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35887 } } }
I20260808 15:55:11.476334 31665 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:11.484664 31660 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:11.484872 31669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:11.490695 31669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:11.511020 31669 catalog_manager.cc:1347] Generated new cluster ID: 4d7fdffe028241388922be389570c19d
I20260808 15:55:11.511276 31669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:11.531649 31669 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:11.533663 31669 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:11.547065 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:11.539426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.27.254.62:35887)
0808 15:55:11.539587 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:11.539593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:11.540293 (+ 700us) server_negotiation.cc:407] Connection header received
0808 15:55:11.540500 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:11.540505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:11.540580 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:11.540690 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:11.541821 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:11.542579 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:11.543868 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:11.544339 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:11.544473 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:11.545618 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:11.545631 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:11.545634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:11.546276 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:11.546340 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:11.546365 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:11.546488 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:11.546620 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 15:55:11.546718 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 15:55:11.558272 31669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf534183ccda47298c7c3a2db41b08ab: Generated new TSK 0
I20260808 15:55:11.559109 31669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:55:11.566680 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31593
I20260808 15:55:11.567948 31655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:11.659435 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31593
2026-08-08T15:55:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1896 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:11 dist-test-slave-srgv krb5kdc[31695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:11 dist-test-slave-srgv krb5kdc[31695](info): set up 2 sockets
Aug 08 15:55:11 dist-test-slave-srgv krb5kdc[31695](info): commencing operation
krb5kdc: starting...
W20260808 15:55:12.756623 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:12 dist-test-slave-srgv krb5kdc[31695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:12.946655 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:43199
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:43199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:13.261382 31708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:13.261693 31708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:13.261881 31708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:13.269091 31708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:13.269214 31708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:13.269312 31708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:13.269441 31708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:13.269527 31708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:13.279561 31708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:43199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:13.280984 31708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:13.282400 31708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:13.294059 31713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:13.294104 31714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:13.294127 31716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:13.295293 31708 server_base.cc:1034] running on GCE node
I20260808 15:55:13.296006 31708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:13.297322 31708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:13.298522 31708 hybrid_clock.cc:648] HybridClock initialized: now 1786204513298484 us; error 35 us; skew 500 ppm
I20260808 15:55:13.298822 31708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:13 dist-test-slave-srgv krb5kdc[31695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:13.301834 31708 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:13.303303 31708 webserver.cc:460] Webserver started at http://127.27.254.62:36983/ using document root <none> and password file <none>
I20260808 15:55:13.303934 31708 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:13.304033 31708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:13.304282 31708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:13.307427 31708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "fca05c894d354ddeaf81f6c1305af81b"
format_stamp: "Formatted at 2026-08-08 15:55:13 on dist-test-slave-srgv"
server_key: "2300f8576aa28d6ad42153f3a8e2013a"
server_key_iv: "6c1c12ef25ce8390713955a84667c440"
server_key_version: "encryptionkey@0"
I20260808 15:55:13.308142 31708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "fca05c894d354ddeaf81f6c1305af81b"
format_stamp: "Formatted at 2026-08-08 15:55:13 on dist-test-slave-srgv"
server_key: "2300f8576aa28d6ad42153f3a8e2013a"
server_key_iv: "6c1c12ef25ce8390713955a84667c440"
server_key_version: "encryptionkey@0"
I20260808 15:55:13.313248 31708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:55:13.316506 31723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:13.317840 31708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:13.317971 31708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "fca05c894d354ddeaf81f6c1305af81b"
format_stamp: "Formatted at 2026-08-08 15:55:13 on dist-test-slave-srgv"
server_key: "2300f8576aa28d6ad42153f3a8e2013a"
server_key_iv: "6c1c12ef25ce8390713955a84667c440"
server_key_version: "encryptionkey@0"
I20260808 15:55:13.318128 31708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:13.334971 31708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:13.338876 31708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:13.339112 31708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:13.357368 31708 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:43199
I20260808 15:55:13.357398 31774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:43199 every 8 connection(s)
I20260808 15:55:13.358848 31708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:13.362413 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:13.364284 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31708
I20260808 15:55:13.364499 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:13.365284 28664 external_mini_cluster.cc:1428] Setting key 092ad27d4088a740fe0b79d982c82b10
I20260808 15:55:13.375034 31775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b: Bootstrap starting.
I20260808 15:55:13.379154 31775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b: Neither blocks nor log segments found. Creating new log.
Aug 08 15:55:13 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:13.380332 31775 log.cc:824] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:13.383996 31775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b: No bootstrap required, opened a new log
I20260808 15:55:13.389835 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:13.367378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.27.254.62:43199)
0808 15:55:13.368209 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:13.368220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:13.368259 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:55:13.369952 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:13.369972 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:13.370363 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:13.370679 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:13.372354 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:13.373748 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:13.375682 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:13.376345 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:13.376479 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:13.381348 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:13.381356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:13.381370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:13.381458 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:13.385265 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:13.385994 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:13.386008 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:13.386017 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:13.386237 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:13.386602 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:13.386607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:13.386611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:13.387346 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:13.387672 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:13.387961 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:55:13.388425 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":394,"threads_started":1}
I20260808 15:55:13.394300 31775 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca05c894d354ddeaf81f6c1305af81b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 43199 } }
I20260808 15:55:13.394747 31775 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:13.394847 31775 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca05c894d354ddeaf81f6c1305af81b, State: Initialized, Role: FOLLOWER
I20260808 15:55:13.395462 31775 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca05c894d354ddeaf81f6c1305af81b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 43199 } }
I20260808 15:55:13.395736 31775 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:13.395913 31775 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:13.396103 31775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:13.398360 31775 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca05c894d354ddeaf81f6c1305af81b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 43199 } }
I20260808 15:55:13.398836 31775 leader_election.cc:302] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca05c894d354ddeaf81f6c1305af81b; no voters:
I20260808 15:55:13.399439 31775 leader_election.cc:288] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:13.399646 31780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:13.401459 31775 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:13.401739 31780 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [term 1 LEADER]: Becoming Leader. State: Replica: fca05c894d354ddeaf81f6c1305af81b, State: Running, Role: LEADER
I20260808 15:55:13.402287 31780 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca05c894d354ddeaf81f6c1305af81b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 43199 } }
I20260808 15:55:13.405165 31782 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca05c894d354ddeaf81f6c1305af81b. Latest consensus state: current_term: 1 leader_uuid: "fca05c894d354ddeaf81f6c1305af81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca05c894d354ddeaf81f6c1305af81b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 43199 } } }
I20260808 15:55:13.405514 31782 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:13.405895 31781 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca05c894d354ddeaf81f6c1305af81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca05c894d354ddeaf81f6c1305af81b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 43199 } } }
I20260808 15:55:13.406298 31781 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:13.407266 31788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:13.412472 31788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:13.421484 31788 catalog_manager.cc:1347] Generated new cluster ID: e3fa4dc0d55c4620ad3b4b8bdcce790e
I20260808 15:55:13.421614 31788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:13.444011 31788 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:13.445950 31788 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:13.457079 31788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca05c894d354ddeaf81f6c1305af81b: Generated new TSK 0
I20260808 15:55:13.458128 31788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:55:13.510948 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:43199
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:13.833245 31801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:13.833563 31801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:13.833766 31801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:13.840723 31801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:13.840821 31801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:13.841019 31801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:13.850502 31801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:13.851722 31801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:13.852986 31801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:13.865947 31806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:13.865981 31809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:13.865899 31807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:13.866619 31801 server_base.cc:1034] running on GCE node
I20260808 15:55:13.867197 31801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:13.868067 31801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:13.869258 31801 hybrid_clock.cc:648] HybridClock initialized: now 1786204513869215 us; error 43 us; skew 500 ppm
I20260808 15:55:13.869622 31801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:13 dist-test-slave-srgv krb5kdc[31695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:13.872650 31801 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:55:13.874194 31801 webserver.cc:460] Webserver started at http://127.27.254.1:40283/ using document root <none> and password file <none>
I20260808 15:55:13.874873 31801 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:13.875020 31801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:13.875325 31801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:13.878249 31801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "99fe2cf861944720a22968ed477cd81c"
format_stamp: "Formatted at 2026-08-08 15:55:13 on dist-test-slave-srgv"
server_key: "ac69f76f28b0b7dcd3c1ae047e2ad40b"
server_key_iv: "816e8a7e0e2b18200ed58dc02d7f2e2a"
server_key_version: "encryptionkey@0"
I20260808 15:55:13.879016 31801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "99fe2cf861944720a22968ed477cd81c"
format_stamp: "Formatted at 2026-08-08 15:55:13 on dist-test-slave-srgv"
server_key: "ac69f76f28b0b7dcd3c1ae047e2ad40b"
server_key_iv: "816e8a7e0e2b18200ed58dc02d7f2e2a"
server_key_version: "encryptionkey@0"
I20260808 15:55:13.883894 31801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:55:13.887208 31816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:13.888415 31801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 15:55:13.888581 31801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "99fe2cf861944720a22968ed477cd81c"
format_stamp: "Formatted at 2026-08-08 15:55:13 on dist-test-slave-srgv"
server_key: "ac69f76f28b0b7dcd3c1ae047e2ad40b"
server_key_iv: "816e8a7e0e2b18200ed58dc02d7f2e2a"
server_key_version: "encryptionkey@0"
I20260808 15:55:13.888789 31801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:13.908286 31801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:13.912513 31801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:13.912823 31801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:13.913985 31801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:13.915328 31801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:13.915429 31801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:13.915580 31801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:13.915652 31801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:13.952519 31801 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:44497
I20260808 15:55:13.952565 31928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:44497 every 8 connection(s)
I20260808 15:55:13.954279 31801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:13.960537 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31801
I20260808 15:55:13.960783 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:13.961490 28664 external_mini_cluster.cc:1428] Setting key 8643dd45029a9df6f9eb842e5400fe21
Aug 08 15:55:13 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204513, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:13.974840 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:13.956792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:42661 (local address 127.27.254.62:43199)
0808 15:55:13.957232 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:13.957238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:13.958401 (+ 1163us) server_negotiation.cc:407] Connection header received
0808 15:55:13.960193 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:13.960200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:13.960294 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:13.960430 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:13.962300 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:13.963148 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:13.964693 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:13.965470 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:13.965607 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:13.968880 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:13.968890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:13.968897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:13.968942 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:13.971623 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:13.972533 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:13.972537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:13.972539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:13.972656 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:13.973127 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:13.973131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:13.973133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:13.973371 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:13.973562 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:13.974429 (+ 867us) server_negotiation.cc:299] Negotiation successful
0808 15:55:13.974586 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":195,"threads_started":1}
I20260808 15:55:13.976475 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:13.957541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:43199 (local address 127.27.254.1:42661)
0808 15:55:13.958240 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:13.958283 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:13.959902 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:13.960615 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:13.960623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:13.961239 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:13.961904 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:13.961922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:13.963310 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:13.963316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:13.964535 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:13.964547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:13.965707 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:13.965713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:13.965897 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:13.966554 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:13.966580 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:13.968654 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:13.971771 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:13.971782 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:13.971801 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:13.972421 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:13.972758 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:13.972765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:13.972771 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:13.973022 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:13.973561 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:13.973571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:13.974009 (+ 438us) client_negotiation.cc:769] Sending connection context
0808 15:55:13.974356 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 15:55:13.974953 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":255,"threads_started":1}
I20260808 15:55:13.978194 31929 heartbeater.cc:347] Connected to a master server at 127.27.254.62:43199
I20260808 15:55:13.978487 31929 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:13.979204 31929 heartbeater.cc:511] Master 127.27.254.62:43199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:55:13.981202 31740 ts_manager.cc:194] Registered new tserver with Master: 99fe2cf861944720a22968ed477cd81c (127.27.254.1:44497)
I20260808 15:55:13.982745 31740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:42661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:14.004814 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:43199
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:14.327099 31935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:14.327363 31935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:14.327514 31935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:14.334731 31935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:14.334828 31935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:14.335038 31935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:14.344417 31935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:14.345674 31935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:14.346972 31935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:14.359575 31941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:14.359596 31940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:14.360172 31943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:14.360415 31935 server_base.cc:1034] running on GCE node
I20260808 15:55:14.361163 31935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:14.362248 31935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:14.363483 31935 hybrid_clock.cc:648] HybridClock initialized: now 1786204514363421 us; error 54 us; skew 500 ppm
I20260808 15:55:14.363817 31935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:14 dist-test-slave-srgv krb5kdc[31695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204514, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:14.372501 31935 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:14.374040 31935 webserver.cc:460] Webserver started at http://127.27.254.2:46129/ using document root <none> and password file <none>
I20260808 15:55:14.374799 31935 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:14.374945 31935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:14.375248 31935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:14.378176 31935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "56e3e7bbe0714cee86cf946ae1e035e6"
format_stamp: "Formatted at 2026-08-08 15:55:14 on dist-test-slave-srgv"
server_key: "63b5d37d0d00b27ba27118207185f624"
server_key_iv: "46e63a3ac8f263e5188b0f3a065c5720"
server_key_version: "encryptionkey@0"
I20260808 15:55:14.378934 31935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "56e3e7bbe0714cee86cf946ae1e035e6"
format_stamp: "Formatted at 2026-08-08 15:55:14 on dist-test-slave-srgv"
server_key: "63b5d37d0d00b27ba27118207185f624"
server_key_iv: "46e63a3ac8f263e5188b0f3a065c5720"
server_key_version: "encryptionkey@0"
I20260808 15:55:14.383939 31935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:14.387333 31950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:14.388609 31935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:14.388775 31935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "56e3e7bbe0714cee86cf946ae1e035e6"
format_stamp: "Formatted at 2026-08-08 15:55:14 on dist-test-slave-srgv"
server_key: "63b5d37d0d00b27ba27118207185f624"
server_key_iv: "46e63a3ac8f263e5188b0f3a065c5720"
server_key_version: "encryptionkey@0"
I20260808 15:55:14.388960 31935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:14.427068 31935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:14.431391 31935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:14.431718 31935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:14.432717 31935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:14.434103 31935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:14.434221 31935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:14.434376 31935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:14.434448 31935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:14.466972 31935 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:39853
I20260808 15:55:14.467027 32064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:39853 every 8 connection(s)
I20260808 15:55:14.468238 31935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:14.472704 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 31935
I20260808 15:55:14.472883 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:14.473505 28664 external_mini_cluster.cc:1428] Setting key 499ff957272a9851885b320a5bafdc0e
Aug 08 15:55:14 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204514, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:14.488381 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:14.470842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:47059 (local address 127.27.254.62:43199)
0808 15:55:14.471024 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:14.471029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:14.472149 (+ 1120us) server_negotiation.cc:407] Connection header received
0808 15:55:14.474687 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:14.474692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:14.474781 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:14.474906 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:14.476553 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.477228 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.478798 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.479208 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.479359 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:14.482803 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:14.482808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:14.482812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:14.482842 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:14.485385 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:14.486141 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:14.486146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:14.486150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:14.486292 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:14.486719 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:14.486724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:14.486726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:14.486995 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:14.487149 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:14.487807 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 15:55:14.488031 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:55:14.489355 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:14.471342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:43199 (local address 127.27.254.2:47059)
0808 15:55:14.472009 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:14.472045 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:14.474407 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:14.475178 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:14.475189 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:14.475851 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:14.476384 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:14.476405 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.477389 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.477394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:14.478630 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:14.478642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.479453 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.479460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:14.479692 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:14.480391 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:14.480414 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:14.482567 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:14.485575 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:14.485584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:14.485599 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:14.486015 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:14.486405 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:14.486410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:14.486414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:14.486595 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:14.487122 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:14.487129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:14.487448 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 15:55:14.487699 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 15:55:14.488216 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
I20260808 15:55:14.490877 32065 heartbeater.cc:347] Connected to a master server at 127.27.254.62:43199
I20260808 15:55:14.491189 32065 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:14.491969 32065 heartbeater.cc:511] Master 127.27.254.62:43199 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
I20260808 15:55:14.493609 31740 ts_manager.cc:194] Registered new tserver with Master: 56e3e7bbe0714cee86cf946ae1e035e6 (127.27.254.2:39853)
I20260808 15:55:14.494412 31740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:47059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:14.514814 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:43199
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:14.831925 32070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:14.832238 32070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:14.832422 32070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:14.839368 32070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:14.839530 32070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:14.839711 32070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:14.849195 32070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:14.850533 32070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:14.851835 32070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:14.864670 32075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:14.864634 32076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:14.864645 32078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:14.865502 32070 server_base.cc:1034] running on GCE node
I20260808 15:55:14.866153 32070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:14.867100 32070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:14.868290 32070 hybrid_clock.cc:648] HybridClock initialized: now 1786204514868236 us; error 49 us; skew 500 ppm
I20260808 15:55:14.868603 32070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:14 dist-test-slave-srgv krb5kdc[31695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204514, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:14.871654 32070 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:14.873230 32070 webserver.cc:460] Webserver started at http://127.27.254.3:45073/ using document root <none> and password file <none>
I20260808 15:55:14.874099 32070 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:14.874229 32070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:14.874549 32070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:14.877516 32070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "c51e1d4d0bd7460c990b4547638d0909"
format_stamp: "Formatted at 2026-08-08 15:55:14 on dist-test-slave-srgv"
server_key: "f82c8f5e4dad6349a451629ea69c8aa1"
server_key_iv: "61e517c0f0d4d665eee3950d1353ce06"
server_key_version: "encryptionkey@0"
I20260808 15:55:14.878307 32070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "c51e1d4d0bd7460c990b4547638d0909"
format_stamp: "Formatted at 2026-08-08 15:55:14 on dist-test-slave-srgv"
server_key: "f82c8f5e4dad6349a451629ea69c8aa1"
server_key_iv: "61e517c0f0d4d665eee3950d1353ce06"
server_key_version: "encryptionkey@0"
I20260808 15:55:14.883318 32070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:14.886693 32085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:14.887948 32070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:55:14.888128 32070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "c51e1d4d0bd7460c990b4547638d0909"
format_stamp: "Formatted at 2026-08-08 15:55:14 on dist-test-slave-srgv"
server_key: "f82c8f5e4dad6349a451629ea69c8aa1"
server_key_iv: "61e517c0f0d4d665eee3950d1353ce06"
server_key_version: "encryptionkey@0"
I20260808 15:55:14.888316 32070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:14.909226 32070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:14.913362 32070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:14.913739 32070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:14.914736 32070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:14.916122 32070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:14.916194 32070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:14.916357 32070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:14.916409 32070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:14.950029 32070 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37991
I20260808 15:55:14.950241 32197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37991 every 8 connection(s)
I20260808 15:55:14.951436 32070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:14.951777 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 32070
I20260808 15:55:14.951958 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:14.952657 28664 external_mini_cluster.cc:1428] Setting key d206a574678749638e7b48b48cb6a08b
Aug 08 15:55:14 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204514, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:14.971901 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:14.956442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:44833 (local address 127.27.254.62:43199)
0808 15:55:14.956596 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:14.956600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:14.956617 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:14.958070 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:14.958075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:14.958161 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:14.958284 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:14.959736 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.960476 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.962062 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.962477 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.962605 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:14.965868 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:14.965874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:14.965879 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:14.965920 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:14.969020 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:14.969799 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:14.969807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:14.969812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:14.969977 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:14.970371 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:14.970374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:14.970376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:14.970661 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:14.970795 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:14.971490 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 15:55:14.971655 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 15:55:14.972880 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:14.955068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:43199 (local address 127.27.254.3:44833)
0808 15:55:14.955978 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:14.956033 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:14.957814 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:14.958516 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:14.958524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:14.959008 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:14.959568 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:14.959584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.960632 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.960637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:14.961916 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:14.961928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:14.962708 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:14.962716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:14.962936 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:14.963509 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:14.963534 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:14.965676 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:14.969143 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:14.969149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:14.969160 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:14.969673 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:14.970088 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:14.970091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:14.970093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:14.970270 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:14.970809 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:14.970816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:14.971147 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 15:55:14.971380 (+ 233us) client_negotiation.cc:240] Negotiation successful
0808 15:55:14.971773 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":340,"threads_started":1}
I20260808 15:55:14.974220 32198 heartbeater.cc:347] Connected to a master server at 127.27.254.62:43199
I20260808 15:55:14.974535 32198 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:14.975265 32198 heartbeater.cc:511] Master 127.27.254.62:43199 requested a full tablet report, sending...
I20260808 15:55:14.976668 31740 ts_manager.cc:194] Registered new tserver with Master: c51e1d4d0bd7460c990b4547638d0909 (127.27.254.3:37991)
I20260808 15:55:14.977571 31740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:44833
I20260808 15:55:14.981287 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:55:14.985620 31929 heartbeater.cc:503] Master 127.27.254.62:43199 was elected leader, sending a full tablet report...
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[31695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:15.029217 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:15.016087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.27.254.62:43199)
0808 15:55:15.016335 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:15.016345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:15.016710 (+ 365us) server_negotiation.cc:407] Connection header received
0808 15:55:15.016866 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:15.016870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:15.016938 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:15.017024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:15.018400 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.019113 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.020299 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.020681 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.020770 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:15.024564 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:15.024569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:15.024572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:15.024602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:15.026933 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.027498 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.027501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.027504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.027622 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.027964 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.027967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.027969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.028270 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:15.028447 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:15.028703 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 15:55:15.028855 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 15:55:15.029822 31740 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50860
I20260808 15:55:15.031812 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31801
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:15.046430 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:15.032785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.27.254.1:44497)
0808 15:55:15.033477 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:15.033484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:15.033507 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:55:15.033559 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:15.033564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:15.033791 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:15.033909 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:15.034933 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.035909 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.037045 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.037642 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.037734 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:15.041501 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:15.041513 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:15.041527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:15.041558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:15.044106 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.044664 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.044671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.044673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.044781 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.045102 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.045105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.045107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.045471 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:15.045688 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:15.045940 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 15:55:15.046135 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":330,"threads_started":1}
W20260808 15:55:15.047637 31924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43970
W20260808 15:55:15.048691 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:55:15.048882 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31801
I20260808 15:55:15.064553 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31935
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:55:15.079932 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:15.065378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.27.254.2:39853)
0808 15:55:15.066005 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:15.066011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:15.066037 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:55:15.066103 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:15.066109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:15.066316 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:15.066441 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:15.067600 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.068694 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.069955 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.070522 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.070624 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:15.074651 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:15.074661 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:15.074675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:15.074714 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:15.077375 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.078060 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.078066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.078069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.078209 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.078530 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.078533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.078535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.078851 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:15.079076 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:15.079426 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 15:55:15.079622 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":319,"threads_started":1}
W20260808 15:55:15.081189 32060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53272
W20260808 15:55:15.082331 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:55:15.082492 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31935
I20260808 15:55:15.097734 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32070
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[31695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:55:15.113814 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:15.098560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.27.254.3:37991)
0808 15:55:15.099175 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:15.099182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:15.099209 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:55:15.099265 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:15.099271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:15.099469 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:15.099597 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:15.100837 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.102032 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.103372 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:15.103916 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:15.104010 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:15.108426 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:15.108437 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:15.108451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:15.108482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:15.111381 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.112028 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.112034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.112037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.112147 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:15.112439 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:15.112442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:15.112444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:15.112754 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:15.112969 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:15.113229 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 15:55:15.113540 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
W20260808 15:55:15.114970 32193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44084
W20260808 15:55:15.115981 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32070: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:55:15.116163 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 32070
I20260808 15:55:15.132719 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 31708
I20260808 15:55:15.133891 31770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:15.234234 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 31708
2026-08-08T15:55:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3580 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[32215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[32215](info): set up 2 sockets
Aug 08 15:55:15 dist-test-slave-srgv krb5kdc[32215](info): commencing operation
krb5kdc: starting...
W20260808 15:55:16.303601 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:16 dist-test-slave-srgv krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:16.497522 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38031
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:38031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:16.811966 32228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:16.812325 32228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:16.812498 32228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:16.819816 32228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:16.819963 32228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:16.820101 32228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:16.820220 32228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:16.820288 32228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:16.829965 32228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:38031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:16.831367 32228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:16.832697 32228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:16.844537 32233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:16.844517 32236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:16.844516 32234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:16.845506 32228 server_base.cc:1034] running on GCE node
I20260808 15:55:16.846343 32228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:16.847740 32228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:16.848989 32228 hybrid_clock.cc:648] HybridClock initialized: now 1786204516848946 us; error 33 us; skew 500 ppm
I20260808 15:55:16.849357 32228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:16 dist-test-slave-srgv krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:16.852434 32228 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:16.854033 32228 webserver.cc:460] Webserver started at http://127.27.254.62:40643/ using document root <none> and password file <none>
I20260808 15:55:16.854729 32228 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:16.854874 32228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:16.855216 32228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:16.858210 32228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "d148972b3f254275b0d8624bbb27f44c"
format_stamp: "Formatted at 2026-08-08 15:55:16 on dist-test-slave-srgv"
server_key: "75743900dd93f25c3eb4496851de3781"
server_key_iv: "4f6c0b0957d42e5fa22a08ca6172ce14"
server_key_version: "encryptionkey@0"
I20260808 15:55:16.858991 32228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "d148972b3f254275b0d8624bbb27f44c"
format_stamp: "Formatted at 2026-08-08 15:55:16 on dist-test-slave-srgv"
server_key: "75743900dd93f25c3eb4496851de3781"
server_key_iv: "4f6c0b0957d42e5fa22a08ca6172ce14"
server_key_version: "encryptionkey@0"
I20260808 15:55:16.864087 32228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:55:16.867499 32243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:16.868921 32228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:16.869131 32228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "d148972b3f254275b0d8624bbb27f44c"
format_stamp: "Formatted at 2026-08-08 15:55:16 on dist-test-slave-srgv"
server_key: "75743900dd93f25c3eb4496851de3781"
server_key_iv: "4f6c0b0957d42e5fa22a08ca6172ce14"
server_key_version: "encryptionkey@0"
I20260808 15:55:16.869335 32228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:16.915942 32228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:16.919926 32228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:16.920238 32228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:16.938689 32228 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:38031
I20260808 15:55:16.938720 32295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:38031 every 8 connection(s)
I20260808 15:55:16.940069 32228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:16.943823 32296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:16.945941 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 32228
I20260808 15:55:16.946173 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:16.947010 28664 external_mini_cluster.cc:1428] Setting key 5f5e132af7b9d876149e63427bf41dab
I20260808 15:55:16.957131 32296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c: Bootstrap starting.
I20260808 15:55:16.961198 32296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c: Neither blocks nor log segments found. Creating new log.
Aug 08 15:55:16 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:16.962419 32296 log.cc:824] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:16.965978 32296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c: No bootstrap required, opened a new log
I20260808 15:55:16.971745 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:16.949141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.27.254.62:38031)
0808 15:55:16.950076 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:16.950087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:16.950129 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:55:16.951832 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:16.951855 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:16.952238 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:16.952546 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:16.954372 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:16.955760 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:16.957621 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:16.958295 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:16.958428 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:16.963246 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:16.963254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:16.963269 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:16.963314 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:16.967071 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:16.967775 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:16.967785 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:16.967800 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:16.968006 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:16.968377 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:16.968382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:16.968385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:16.969110 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:16.969485 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:16.969866 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 15:55:16.970360 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":483,"threads_started":1}
I20260808 15:55:16.976341 32296 raft_consensus.cc:348] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d148972b3f254275b0d8624bbb27f44c" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38031 } }
I20260808 15:55:16.976794 32296 raft_consensus.cc:374] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:16.976908 32296 raft_consensus.cc:729] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d148972b3f254275b0d8624bbb27f44c, State: Initialized, Role: FOLLOWER
I20260808 15:55:16.977563 32296 consensus_queue.cc:260] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d148972b3f254275b0d8624bbb27f44c" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38031 } }
I20260808 15:55:16.977818 32296 raft_consensus.cc:388] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:16.977946 32296 raft_consensus.cc:482] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:16.978132 32296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:16.980294 32296 raft_consensus.cc:504] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d148972b3f254275b0d8624bbb27f44c" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38031 } }
I20260808 15:55:16.980822 32296 leader_election.cc:302] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d148972b3f254275b0d8624bbb27f44c; no voters:
I20260808 15:55:16.981381 32296 leader_election.cc:288] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:16.981585 32301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:16.982362 32301 raft_consensus.cc:686] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [term 1 LEADER]: Becoming Leader. State: Replica: d148972b3f254275b0d8624bbb27f44c, State: Running, Role: LEADER
I20260808 15:55:16.983022 32301 consensus_queue.cc:237] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d148972b3f254275b0d8624bbb27f44c" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38031 } }
I20260808 15:55:16.983672 32296 sys_catalog.cc:564] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:16.986209 32303 sys_catalog.cc:455] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d148972b3f254275b0d8624bbb27f44c. Latest consensus state: current_term: 1 leader_uuid: "d148972b3f254275b0d8624bbb27f44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d148972b3f254275b0d8624bbb27f44c" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38031 } } }
I20260808 15:55:16.986589 32303 sys_catalog.cc:458] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:16.987169 32302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d148972b3f254275b0d8624bbb27f44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d148972b3f254275b0d8624bbb27f44c" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38031 } } }
I20260808 15:55:16.987515 32302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:16.987751 32308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:16.995739 32308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:17.008827 32308 catalog_manager.cc:1347] Generated new cluster ID: 828b783f19a7496497bc28bd2426861c
I20260808 15:55:17.008965 32308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:17.038916 32308 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:17.040227 32308 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:17.059223 32308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d148972b3f254275b0d8624bbb27f44c: Generated new TSK 0
I20260808 15:55:17.060305 32308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:55:17.108821 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38031
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:17.432919 32323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:17.433182 32323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:17.433339 32323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:17.440328 32323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:17.440428 32323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:17.440622 32323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:17.450201 32323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:17.451507 32323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:17.452817 32323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:17.465937 32328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:17.465916 32329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:17.465992 32331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:17.467145 32323 server_base.cc:1034] running on GCE node
I20260808 15:55:17.467779 32323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:17.468698 32323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:17.469924 32323 hybrid_clock.cc:648] HybridClock initialized: now 1786204517469880 us; error 57 us; skew 500 ppm
I20260808 15:55:17.470361 32323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:17 dist-test-slave-srgv krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:17.473714 32323 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:55:17.475164 32323 webserver.cc:460] Webserver started at http://127.27.254.1:33849/ using document root <none> and password file <none>
I20260808 15:55:17.475795 32323 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:17.475890 32323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:17.476163 32323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:17.479100 32323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "60368d3d4a4c453da7ad972bd8b9b5f7"
format_stamp: "Formatted at 2026-08-08 15:55:17 on dist-test-slave-srgv"
server_key: "054642707dbbda3d72b66951b439bc7e"
server_key_iv: "81268e8789a92b1366ac252d6df166c8"
server_key_version: "encryptionkey@0"
I20260808 15:55:17.479836 32323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "60368d3d4a4c453da7ad972bd8b9b5f7"
format_stamp: "Formatted at 2026-08-08 15:55:17 on dist-test-slave-srgv"
server_key: "054642707dbbda3d72b66951b439bc7e"
server_key_iv: "81268e8789a92b1366ac252d6df166c8"
server_key_version: "encryptionkey@0"
I20260808 15:55:17.485153 32323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:55:17.488601 32338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:17.489943 32323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:17.490129 32323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "60368d3d4a4c453da7ad972bd8b9b5f7"
format_stamp: "Formatted at 2026-08-08 15:55:17 on dist-test-slave-srgv"
server_key: "054642707dbbda3d72b66951b439bc7e"
server_key_iv: "81268e8789a92b1366ac252d6df166c8"
server_key_version: "encryptionkey@0"
I20260808 15:55:17.490324 32323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:17.505811 32323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:17.509943 32323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:17.510242 32323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:17.511152 32323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:17.512425 32323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:17.512531 32323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:17.512701 32323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:17.512778 32323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:17.545902 32323 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:39095
I20260808 15:55:17.545946 32450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:39095 every 8 connection(s)
I20260808 15:55:17.547223 32323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:17.548739 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 32323
I20260808 15:55:17.548967 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:17.549678 28664 external_mini_cluster.cc:1428] Setting key 2f6c685a5791f017589c437b9e139654
Aug 08 15:55:17 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:55:17.568933 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:17.549932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:37103 (local address 127.27.254.62:38031)
0808 15:55:17.550445 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:17.550453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:17.551557 (+ 1104us) server_negotiation.cc:407] Connection header received
0808 15:55:17.553454 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:17.553462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:17.553549 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:17.553683 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:17.555104 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:17.556272 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:17.558575 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:17.559022 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:17.559119 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:17.563281 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:17.563286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:17.563289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:17.563321 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:17.565893 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:17.566610 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:17.566618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:17.566621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:17.566748 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:17.567215 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:17.567220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:17.567224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:17.567487 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:17.567642 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:17.568324 (+ 682us) server_negotiation.cc:299] Negotiation successful
0808 15:55:17.568556 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260808 15:55:17.570103 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:17.550481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38031 (local address 127.27.254.1:37103)
0808 15:55:17.551380 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:17.551440 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:17.553165 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:17.553867 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:17.553874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:17.554377 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:17.554934 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:17.554951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:17.556430 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:17.556434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:17.558434 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:17.558452 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:17.559240 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:17.559249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:17.559588 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:17.560314 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:17.560347 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:17.563032 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:17.566031 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:17.566040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:17.566057 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:17.566485 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:17.566861 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:17.566866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:17.566870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:17.567073 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:17.567616 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:17.567624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:17.567976 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 15:55:17.568247 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 15:55:17.568806 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
I20260808 15:55:17.571766 32451 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38031
I20260808 15:55:17.572124 32451 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:17.572916 32451 heartbeater.cc:511] Master 127.27.254.62:38031 requested a full tablet report, sending...
I20260808 15:55:17.575317 32261 ts_manager.cc:194] Registered new tserver with Master: 60368d3d4a4c453da7ad972bd8b9b5f7 (127.27.254.1:39095)
I20260808 15:55:17.577457 32261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:37103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:17.594604 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38031
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:17.910022 32457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:17.910338 32457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:17.910432 32457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:17.917608 32457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:17.917707 32457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:17.917815 32457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:17.927372 32457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:17.928540 32457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:17.929849 32457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:17.942853 32463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:17.943172 32465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:17.942884 32462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:17.944787 32457 server_base.cc:1034] running on GCE node
I20260808 15:55:17.945490 32457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:17.946732 32457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:17.952603 32457 hybrid_clock.cc:648] HybridClock initialized: now 1786204517952495 us; error 113 us; skew 500 ppm
I20260808 15:55:17.953022 32457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:17 dist-test-slave-srgv krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:17.957037 32457 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:17.959153 32457 webserver.cc:460] Webserver started at http://127.27.254.2:41217/ using document root <none> and password file <none>
I20260808 15:55:17.960074 32457 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:17.960237 32457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:17.960595 32457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:17.965039 32457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "f2b186a889174bc29a36c4069e5f3a83"
format_stamp: "Formatted at 2026-08-08 15:55:17 on dist-test-slave-srgv"
server_key: "985bef725e8c4f19352bfff456ce2523"
server_key_iv: "1ea60d1d2f2978d6742a95fb49b2e1c6"
server_key_version: "encryptionkey@0"
I20260808 15:55:17.966215 32457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "f2b186a889174bc29a36c4069e5f3a83"
format_stamp: "Formatted at 2026-08-08 15:55:17 on dist-test-slave-srgv"
server_key: "985bef725e8c4f19352bfff456ce2523"
server_key_iv: "1ea60d1d2f2978d6742a95fb49b2e1c6"
server_key_version: "encryptionkey@0"
I20260808 15:55:17.973663 32457 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260808 15:55:17.978483 32472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:17.980154 32457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260808 15:55:17.980314 32457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "f2b186a889174bc29a36c4069e5f3a83"
format_stamp: "Formatted at 2026-08-08 15:55:17 on dist-test-slave-srgv"
server_key: "985bef725e8c4f19352bfff456ce2523"
server_key_iv: "1ea60d1d2f2978d6742a95fb49b2e1c6"
server_key_version: "encryptionkey@0"
I20260808 15:55:17.980537 32457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:17.992789 32457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:17.996609 32457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:17.996902 32457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:17.997840 32457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:17.999086 32457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:17.999187 32457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:17.999328 32457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:17.999399 32457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:18.032509 32457 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:41421
I20260808 15:55:18.032559 32584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:41421 every 8 connection(s)
I20260808 15:55:18.033831 32457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:18.034252 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 32457
I20260808 15:55:18.034474 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:18.035081 28664 external_mini_cluster.cc:1428] Setting key b271c55874a665331f01d5de7ce40f09
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204517, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
I20260808 15:55:18.055023 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.036647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:50153 (local address 127.27.254.62:38031)
0808 15:55:18.036816 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.036820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.038299 (+ 1479us) server_negotiation.cc:407] Connection header received
0808 15:55:18.040090 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.040099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.040191 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.040304 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.041968 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.042816 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.044526 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.045292 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.045445 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.049082 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.049087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.049091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.049129 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.051899 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.052622 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.052627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.052636 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.052769 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.053192 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.053197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.053205 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.053554 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.053723 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.054443 (+ 720us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.054679 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
I20260808 15:55:18.056188 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.037245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38031 (local address 127.27.254.2:50153)
0808 15:55:18.038100 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:18.038176 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:18.039821 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:18.040495 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:18.040503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:18.041153 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:18.041803 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:18.041821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.042974 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.042981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:18.044382 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:18.044393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.045537 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.045543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:18.045744 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.046474 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:18.046496 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:18.048870 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:18.052038 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:18.052047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:18.052063 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:18.052489 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:18.052879 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:18.052884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:18.052888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:18.053072 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:18.053694 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:18.053702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:18.054050 (+ 348us) client_negotiation.cc:769] Sending connection context
0808 15:55:18.054316 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:55:18.054863 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":371,"threads_started":1}
I20260808 15:55:18.057723 32585 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38031
I20260808 15:55:18.058060 32585 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:18.058925 32585 heartbeater.cc:511] Master 127.27.254.62:38031 requested a full tablet report, sending...
I20260808 15:55:18.060570 32261 ts_manager.cc:194] Registered new tserver with Master: f2b186a889174bc29a36c4069e5f3a83 (127.27.254.2:41421)
I20260808 15:55:18.061385 32261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:50153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:18.078639 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38031
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:18.399530 32590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:18.399784 32590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:18.399880 32590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:18.407037 32590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:18.407136 32590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:18.407251 32590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:18.416692 32590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:18.417836 32590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:18.419066 32590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:18.431715 32595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:18.431692 32596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:18.431915 32598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:18.432245 32590 server_base.cc:1034] running on GCE node
I20260808 15:55:18.432938 32590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:18.433868 32590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:18.435042 32590 hybrid_clock.cc:648] HybridClock initialized: now 1786204518434994 us; error 44 us; skew 500 ppm
I20260808 15:55:18.435318 32590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:18.438405 32590 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:18.439816 32590 webserver.cc:460] Webserver started at http://127.27.254.3:46881/ using document root <none> and password file <none>
I20260808 15:55:18.440495 32590 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:18.440591 32590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:18.440925 32590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:18.443720 32590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "266bee5c3145423f89ad34d6963dfdef"
format_stamp: "Formatted at 2026-08-08 15:55:18 on dist-test-slave-srgv"
server_key: "50cd45e1ee5d7e2cb2ea7742a8a5ae7f"
server_key_iv: "c11f64f1e2a89a0b35a6ac46d9f98221"
server_key_version: "encryptionkey@0"
I20260808 15:55:18.444430 32590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "266bee5c3145423f89ad34d6963dfdef"
format_stamp: "Formatted at 2026-08-08 15:55:18 on dist-test-slave-srgv"
server_key: "50cd45e1ee5d7e2cb2ea7742a8a5ae7f"
server_key_iv: "c11f64f1e2a89a0b35a6ac46d9f98221"
server_key_version: "encryptionkey@0"
I20260808 15:55:18.449172 32590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260808 15:55:18.452569 32605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:18.453919 32590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:18.454084 32590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "266bee5c3145423f89ad34d6963dfdef"
format_stamp: "Formatted at 2026-08-08 15:55:18 on dist-test-slave-srgv"
server_key: "50cd45e1ee5d7e2cb2ea7742a8a5ae7f"
server_key_iv: "c11f64f1e2a89a0b35a6ac46d9f98221"
server_key_version: "encryptionkey@0"
I20260808 15:55:18.454283 32590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:18.471217 32590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:18.475124 32590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:18.475407 32590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:18.476405 32590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:18.477728 32590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:18.477828 32590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:18.477978 32590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:18.478066 32590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:18.511672 32590 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:45937
I20260808 15:55:18.511729 32717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:45937 every 8 connection(s)
I20260808 15:55:18.512943 32590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:18.517138 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 32590
I20260808 15:55:18.517387 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:18.517874 28664 external_mini_cluster.cc:1428] Setting key 7ae76fcbc477540698c05d68828f8455
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:18.532279 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.515472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:54951 (local address 127.27.254.62:38031)
0808 15:55:18.515656 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.515661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.516952 (+ 1291us) server_negotiation.cc:407] Connection header received
0808 15:55:18.519030 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.519035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.519125 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.519240 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.520710 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.521506 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.522983 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.523394 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.523522 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.526815 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.526820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.526823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.526854 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.529183 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.529947 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.529954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.529960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.530107 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.530548 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.530553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.530555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.530833 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.530988 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.531665 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.531926 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:55:18.533346 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.515930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38031 (local address 127.27.254.3:54951)
0808 15:55:18.516796 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:18.516838 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:18.518752 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:18.519485 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:18.519493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:18.519975 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:18.520537 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:18.520553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.521656 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.521661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:18.522845 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:18.522855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.523612 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.523620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:18.523852 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.524436 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:18.524460 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:18.526607 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:18.529324 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:18.529333 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:18.529348 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:18.529829 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:18.530229 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:18.530234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:18.530238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:18.530421 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:18.530972 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:18.530979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:18.531315 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 15:55:18.531575 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 15:55:18.532101 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":428,"threads_started":1}
I20260808 15:55:18.534873 32718 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38031
I20260808 15:55:18.535161 32718 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:18.535945 32718 heartbeater.cc:511] Master 127.27.254.62:38031 requested a full tablet report, sending...
I20260808 15:55:18.537289 32261 ts_manager.cc:194] Registered new tserver with Master: 266bee5c3145423f89ad34d6963dfdef (127.27.254.3:45937)
I20260808 15:55:18.538108 32261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:54951
I20260808 15:55:18.545164 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260808 15:55:18.580391 32451 heartbeater.cc:503] Master 127.27.254.62:38031 was elected leader, sending a full tablet report...
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:18.591789 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.577191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.27.254.1:39095)
0808 15:55:18.577827 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.577834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.577859 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:55:18.577917 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.577923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.578133 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.578284 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.579614 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.580909 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.582267 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.582792 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.582884 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.586743 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.586753 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.586767 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.586797 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.589381 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.590011 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.590018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.590020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.590147 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.590478 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.590481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.590483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.590798 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.591013 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.591224 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.591431 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":375,"threads_started":1}
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:18.608741 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.596119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.27.254.62:38031)
0808 15:55:18.596348 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.596353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.596469 (+ 116us) server_negotiation.cc:407] Connection header received
0808 15:55:18.596588 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.596592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.596652 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.596731 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.597936 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.598677 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.599908 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.600358 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.600453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.604143 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.604147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.604150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.604180 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.606591 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.607126 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.607130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.607132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.607244 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.607525 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.607527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.607529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.607830 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.608001 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.608243 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.608396 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 15:55:18.609268 32261 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46428
I20260808 15:55:18.610334 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32323
I20260808 15:55:18.623196 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.611259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.27.254.1:39095)
0808 15:55:18.611461 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.611468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.611627 (+ 159us) server_negotiation.cc:407] Connection header received
0808 15:55:18.611817 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.611829 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.611891 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.611990 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.613129 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.613881 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.615038 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.615459 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.615589 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.617850 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.617855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.617858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.617886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.620670 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.621337 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.621344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.621347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.621513 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.621887 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.621893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.621897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.622135 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.622378 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.622664 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.622874 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:55:18.623833 32446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:18.707278 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 32323
I20260808 15:55:18.723112 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32457
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:55:18.738492 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.723986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.27.254.2:41421)
0808 15:55:18.724505 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.724512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.724536 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:55:18.724599 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.724604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.724820 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.724946 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.726191 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.727212 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.728438 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.728970 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.729063 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.733141 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.733153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.733167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.733207 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.735824 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.736452 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.736461 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.736463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.736631 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.736951 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.736954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.736956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.737289 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.737635 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.737966 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.738187 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":268,"threads_started":1}
I20260808 15:55:18.739748 32580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:18.823395 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 32457
I20260808 15:55:18.839027 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32590
Aug 08 15:55:18 dist-test-slave-srgv krb5kdc[32215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:55:18.855453 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:18.839981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.27.254.3:45937)
0808 15:55:18.840744 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:18.840751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:18.840778 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:55:18.840840 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:18.840846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:18.841047 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:18.841190 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:18.842701 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.843682 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.844911 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:18.845567 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:18.845667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:18.850185 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:18.850196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:18.850210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:18.850242 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:18.852974 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.853700 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.853707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.853710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.853835 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:18.854174 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:18.854177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:18.854179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:18.854500 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:18.854728 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:18.854973 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 15:55:18.855170 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":444,"threads_started":1}
I20260808 15:55:18.856750 32713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:18.939155 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 32590
I20260808 15:55:18.957480 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32228
I20260808 15:55:18.958632 32291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:19.059718 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 32228
2026-08-08T15:55:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3825 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:19 dist-test-slave-srgv krb5kdc[32737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:19 dist-test-slave-srgv krb5kdc[32737](info): set up 2 sockets
Aug 08 15:55:19 dist-test-slave-srgv krb5kdc[32737](info): commencing operation
krb5kdc: starting...
W20260808 15:55:20.139268 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:20 dist-test-slave-srgv krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:20.332623 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37147
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:20.642637 32750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:20.642970 32750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:20.643138 32750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:20.650094 32750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:20.650235 32750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:20.650344 32750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:20.650449 32750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:20.650540 32750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:20.659951 32750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:20.661243 32750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:20.662547 32750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:20.674001 32756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:20.674032 32758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:20.674216 32755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:20.674804 32750 server_base.cc:1034] running on GCE node
I20260808 15:55:20.675488 32750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:20.676725 32750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:20.677939 32750 hybrid_clock.cc:648] HybridClock initialized: now 1786204520677901 us; error 53 us; skew 500 ppm
I20260808 15:55:20.678275 32750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:20 dist-test-slave-srgv krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204520, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:20.689169 32750 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:20.690439 32750 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:20.690568 32750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:20.690897 32750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:20.693799 32750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "24f264b039d54391b07d407ffac506e0"
format_stamp: "Formatted at 2026-08-08 15:55:20 on dist-test-slave-srgv"
server_key: "b38eb531da1546cb5dfe782a570f0458"
server_key_iv: "dada9857145edd6f24893c406b3d35c7"
server_key_version: "encryptionkey@0"
I20260808 15:55:20.694590 32750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "24f264b039d54391b07d407ffac506e0"
format_stamp: "Formatted at 2026-08-08 15:55:20 on dist-test-slave-srgv"
server_key: "b38eb531da1546cb5dfe782a570f0458"
server_key_iv: "dada9857145edd6f24893c406b3d35c7"
server_key_version: "encryptionkey@0"
I20260808 15:55:20.699492 32750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:20.702836 32764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:20.704064 32750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:55:20.704254 32750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "24f264b039d54391b07d407ffac506e0"
format_stamp: "Formatted at 2026-08-08 15:55:20 on dist-test-slave-srgv"
server_key: "b38eb531da1546cb5dfe782a570f0458"
server_key_iv: "dada9857145edd6f24893c406b3d35c7"
server_key_version: "encryptionkey@0"
I20260808 15:55:20.704475 32750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:20.729977 32750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:20.733935 32750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:20.734208 32750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:20.752439 32750 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:37147
I20260808 15:55:20.752460 347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:37147 every 8 connection(s)
I20260808 15:55:20.753341 32750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:20.756892 348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:20.760026 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 32750
I20260808 15:55:20.760356 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:20.760969 28664 external_mini_cluster.cc:1428] Setting key 99a49f1bf03f6ce177d452007d252e72
I20260808 15:55:20.770032 348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0: Bootstrap starting.
I20260808 15:55:20.773772 348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:20.774750 348 log.cc:824] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:55:20 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:20.777894 348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0: No bootstrap required, opened a new log
I20260808 15:55:20.785848 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:20.763029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.27.254.62:37147)
0808 15:55:20.763825 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:20.763841 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:20.763892 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:55:20.765813 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:20.765847 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:20.766368 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:20.766815 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:20.768192 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:20.770131 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:20.771459 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:20.772551 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:20.772734 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:20.776611 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:20.776622 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:20.776640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:20.776685 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:20.781132 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:20.781921 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:20.781930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:20.781939 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:20.782157 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:20.782527 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:20.782533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:20.782536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:20.783245 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:20.783539 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:20.783907 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 15:55:20.784395 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":342,"threads_started":1}
I20260808 15:55:20.788515 348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f264b039d54391b07d407ffac506e0" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37147 } }
I20260808 15:55:20.788977 348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:20.789096 348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f264b039d54391b07d407ffac506e0, State: Initialized, Role: FOLLOWER
I20260808 15:55:20.789623 348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f264b039d54391b07d407ffac506e0" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37147 } }
I20260808 15:55:20.789832 348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:20.789929 348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:20.790064 348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:20.792533 348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f264b039d54391b07d407ffac506e0" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37147 } }
I20260808 15:55:20.792997 348 leader_election.cc:302] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f264b039d54391b07d407ffac506e0; no voters:
I20260808 15:55:20.793574 348 leader_election.cc:288] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:20.793732 353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:20.794518 353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [term 1 LEADER]: Becoming Leader. State: Replica: 24f264b039d54391b07d407ffac506e0, State: Running, Role: LEADER
I20260808 15:55:20.795069 353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f264b039d54391b07d407ffac506e0" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37147 } }
I20260808 15:55:20.795743 348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:20.798301 354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24f264b039d54391b07d407ffac506e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f264b039d54391b07d407ffac506e0" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37147 } } }
I20260808 15:55:20.798302 355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f264b039d54391b07d407ffac506e0. Latest consensus state: current_term: 1 leader_uuid: "24f264b039d54391b07d407ffac506e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f264b039d54391b07d407ffac506e0" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37147 } } }
I20260808 15:55:20.798877 355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:20.799664 354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:20.806883 359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:20.813719 359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:20.824849 359 catalog_manager.cc:1347] Generated new cluster ID: 2692ca1d726d48b9b37fff659e726843
I20260808 15:55:20.824975 359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:20.832541 359 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:20.833904 359 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:20.848157 359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0: Generated new TSK 0
I20260808 15:55:20.849007 359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:55:20.908320 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37147
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:21.237787 374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:21.238099 374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:21.238274 374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:21.245466 374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:21.245572 374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:21.245779 374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:21.255498 374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:21.256803 374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:21.258239 374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:21.271688 379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:21.271698 382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:21.271615 380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:21.272409 374 server_base.cc:1034] running on GCE node
I20260808 15:55:21.273056 374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:21.274048 374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:21.275251 374 hybrid_clock.cc:648] HybridClock initialized: now 1786204521275185 us; error 54 us; skew 500 ppm
I20260808 15:55:21.275573 374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:21 dist-test-slave-srgv krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:21.278760 374 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:55:21.279932 374 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:21.280056 374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:21.280377 374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:21.283263 374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "3aec8f11262043f8bf3c9a4689b0bbac"
format_stamp: "Formatted at 2026-08-08 15:55:21 on dist-test-slave-srgv"
server_key: "8db3831deaf8eb5bc461c143435850ef"
server_key_iv: "bb72ef84eaea4735ea64fd833378f582"
server_key_version: "encryptionkey@0"
I20260808 15:55:21.284003 374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "3aec8f11262043f8bf3c9a4689b0bbac"
format_stamp: "Formatted at 2026-08-08 15:55:21 on dist-test-slave-srgv"
server_key: "8db3831deaf8eb5bc461c143435850ef"
server_key_iv: "bb72ef84eaea4735ea64fd833378f582"
server_key_version: "encryptionkey@0"
I20260808 15:55:21.288859 374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:55:21.292194 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:21.293574 374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:21.293736 374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "3aec8f11262043f8bf3c9a4689b0bbac"
format_stamp: "Formatted at 2026-08-08 15:55:21 on dist-test-slave-srgv"
server_key: "8db3831deaf8eb5bc461c143435850ef"
server_key_iv: "bb72ef84eaea4735ea64fd833378f582"
server_key_version: "encryptionkey@0"
I20260808 15:55:21.293977 374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:21.307602 374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:21.311781 374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:21.312052 374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:21.312948 374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:21.314360 374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:21.314477 374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:21.314616 374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:21.314702 374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:21.347335 374 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:40041
I20260808 15:55:21.347438 500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:40041 every 8 connection(s)
I20260808 15:55:21.348333 374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:21.349911 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 374
I20260808 15:55:21.350188 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:21.350929 28664 external_mini_cluster.cc:1428] Setting key a799a937c0d2c171ee4beb6969727ac5
Aug 08 15:55:21 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:21.371465 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:21.350911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:43535 (local address 127.27.254.62:37147)
0808 15:55:21.351360 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:21.351367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:21.352567 (+ 1200us) server_negotiation.cc:407] Connection header received
0808 15:55:21.355204 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:21.355209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:21.355273 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:21.355370 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:21.357503 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.358225 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.360211 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.360607 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.360744 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:21.364380 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:21.364390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:21.364397 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:21.364444 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:21.368092 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:21.369067 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:21.369072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:21.369074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:21.369196 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:21.369712 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:21.369716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:21.369718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:21.369973 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:21.370096 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:21.371020 (+ 924us) server_negotiation.cc:299] Negotiation successful
0808 15:55:21.371192 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":212,"threads_started":1}
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:55:21.372941 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:21.351517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37147 (local address 127.27.254.1:43535)
0808 15:55:21.352381 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:21.352445 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:21.354863 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:21.355642 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:21.355657 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:21.356439 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:21.357260 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:21.357288 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.358382 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.358391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:21.360045 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:21.360058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.360837 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.360846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:21.361059 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:21.361774 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:21.361800 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:21.364143 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:21.368233 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:21.368239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:21.368252 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:21.368947 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:21.369313 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:21.369320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:21.369326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:21.369600 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:21.370117 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:21.370128 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:21.370596 (+ 468us) client_negotiation.cc:769] Sending connection context
0808 15:55:21.370933 (+ 337us) client_negotiation.cc:240] Negotiation successful
0808 15:55:21.371502 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":353,"threads_started":1}
I20260808 15:55:21.374503 501 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37147
I20260808 15:55:21.374846 501 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
I20260808 15:55:21.375535 501 heartbeater.cc:511] Master 127.27.254.62:37147 requested a full tablet report, sending...
I20260808 15:55:21.377831 313 ts_manager.cc:194] Registered new tserver with Master: 3aec8f11262043f8bf3c9a4689b0bbac (127.27.254.1:40041)
I20260808 15:55:21.379992 313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:43535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:21.399040 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37147
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:21.716441 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:21.716743 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:21.716917 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:21.724160 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:21.724315 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:21.724489 507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:21.734136 507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:21.735410 507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:21.736714 507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:21.749709 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:21.749732 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:21.749648 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:21.750480 507 server_base.cc:1034] running on GCE node
I20260808 15:55:21.751165 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:21.752421 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:21.753655 507 hybrid_clock.cc:648] HybridClock initialized: now 1786204521753597 us; error 45 us; skew 500 ppm
I20260808 15:55:21.754057 507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:21 dist-test-slave-srgv krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:21.758818 507 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:21.760176 507 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:21.760304 507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:21.760624 507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:21.765286 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "85fae41340b141148781fa2a32630b1a"
format_stamp: "Formatted at 2026-08-08 15:55:21 on dist-test-slave-srgv"
server_key: "d532acb2df0238c971a3364b7c402307"
server_key_iv: "86624212e3e962cee3cecb8caea8aa62"
server_key_version: "encryptionkey@0"
I20260808 15:55:21.766732 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "85fae41340b141148781fa2a32630b1a"
format_stamp: "Formatted at 2026-08-08 15:55:21 on dist-test-slave-srgv"
server_key: "d532acb2df0238c971a3364b7c402307"
server_key_iv: "86624212e3e962cee3cecb8caea8aa62"
server_key_version: "encryptionkey@0"
I20260808 15:55:21.773790 507 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 15:55:21.778658 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:21.780421 507 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:55:21.780592 507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "85fae41340b141148781fa2a32630b1a"
format_stamp: "Formatted at 2026-08-08 15:55:21 on dist-test-slave-srgv"
server_key: "d532acb2df0238c971a3364b7c402307"
server_key_iv: "86624212e3e962cee3cecb8caea8aa62"
server_key_version: "encryptionkey@0"
I20260808 15:55:21.780839 507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:21.820051 507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:21.823808 507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:21.824034 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:21.824896 507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:21.826113 507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:21.826197 507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:21.826269 507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:21.826321 507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:21.859716 507 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:41075
I20260808 15:55:21.859776 633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:41075 every 8 connection(s)
I20260808 15:55:21.860565 507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:21.870769 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 507
I20260808 15:55:21.870980 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:21.871588 28664 external_mini_cluster.cc:1428] Setting key ff188698f52812e35b891c61566a092d
Aug 08 15:55:21 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204521, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:21.881286 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:21.863005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:33737 (local address 127.27.254.62:37147)
0808 15:55:21.863183 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:21.863187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:21.864292 (+ 1105us) server_negotiation.cc:407] Connection header received
0808 15:55:21.866158 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:21.866163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:21.866260 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:21.866371 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:21.867863 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.868583 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.870242 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.870694 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.870805 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:21.874572 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:21.874582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:21.874589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:21.874634 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:21.878156 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:21.878862 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:21.878869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:21.878875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:21.879042 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:21.879448 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:21.879456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:21.879462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:21.879857 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:21.879990 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:21.880892 (+ 902us) server_negotiation.cc:299] Negotiation successful
0808 15:55:21.881051 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:55:21.882953 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:21.863446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37147 (local address 127.27.254.2:33737)
0808 15:55:21.864123 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:21.864165 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:21.865890 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:21.866592 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:21.866600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:21.867111 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:21.867676 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:21.867702 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.868740 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.868745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:21.870082 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:21.870106 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:21.870925 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:21.870934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:21.871217 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:21.871988 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:21.872010 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:21.874357 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:21.878285 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:21.878292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:21.878303 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:21.878731 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:21.879154 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:21.879157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:21.879159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:21.879331 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:21.880011 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:21.880021 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:21.880468 (+ 447us) client_negotiation.cc:769] Sending connection context
0808 15:55:21.880810 (+ 342us) client_negotiation.cc:240] Negotiation successful
0808 15:55:21.881398 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":279,"threads_started":1}
I20260808 15:55:21.884418 634 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37147
I20260808 15:55:21.884728 634 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:21.885385 634 heartbeater.cc:511] Master 127.27.254.62:37147 requested a full tablet report, sending...
I20260808 15:55:21.886883 313 ts_manager.cc:194] Registered new tserver with Master: 85fae41340b141148781fa2a32630b1a (127.27.254.2:41075)
I20260808 15:55:21.887775 313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:33737
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:21.916818 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37147
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:22.241295 639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:22.241683 639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:22.241873 639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:22.249042 639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:22.249200 639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:22.249396 639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:22.259111 639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:37147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:22.260462 639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:22.261857 639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:22.274225 644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:22.274271 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:22.274178 645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:22.275032 639 server_base.cc:1034] running on GCE node
I20260808 15:55:22.275530 639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:22.276376 639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:22.277561 639 hybrid_clock.cc:648] HybridClock initialized: now 1786204522277504 us; error 50 us; skew 500 ppm
I20260808 15:55:22.277884 639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:22 dist-test-slave-srgv krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:22.280830 639 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:22.281950 639 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:22.282037 639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:22.282291 639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:22.285005 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "81ace5453ad24984b449ad78f3c8b43f"
format_stamp: "Formatted at 2026-08-08 15:55:22 on dist-test-slave-srgv"
server_key: "9cb557c465ba3c6f7d3bfd72d696e36a"
server_key_iv: "f41eedf3809d225f09ab436c6a5f44a9"
server_key_version: "encryptionkey@0"
I20260808 15:55:22.285833 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "81ace5453ad24984b449ad78f3c8b43f"
format_stamp: "Formatted at 2026-08-08 15:55:22 on dist-test-slave-srgv"
server_key: "9cb557c465ba3c6f7d3bfd72d696e36a"
server_key_iv: "f41eedf3809d225f09ab436c6a5f44a9"
server_key_version: "encryptionkey@0"
I20260808 15:55:22.290633 639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:55:22.293804 653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:22.295022 639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:55:22.295185 639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "81ace5453ad24984b449ad78f3c8b43f"
format_stamp: "Formatted at 2026-08-08 15:55:22 on dist-test-slave-srgv"
server_key: "9cb557c465ba3c6f7d3bfd72d696e36a"
server_key_iv: "f41eedf3809d225f09ab436c6a5f44a9"
server_key_version: "encryptionkey@0"
I20260808 15:55:22.295396 639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:22.311985 639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:22.315912 639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:22.316187 639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:22.317116 639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:22.318495 639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:22.318615 639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:22.318756 639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:22.318846 639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:22.352035 639 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37901
I20260808 15:55:22.352072 765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37901 every 8 connection(s)
I20260808 15:55:22.352910 639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:22.354974 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 639
I20260808 15:55:22.355208 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:22.355686 28664 external_mini_cluster.cc:1428] Setting key b69f7dee4f9016455711d758fcbcc940
Aug 08 15:55:22 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204522, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:22.372160 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.355315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:43233 (local address 127.27.254.62:37147)
0808 15:55:22.355526 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.355532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.356733 (+ 1201us) server_negotiation.cc:407] Connection header received
0808 15:55:22.358520 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.358525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.358648 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.358757 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:22.360196 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.360921 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.362587 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.362991 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.363123 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.366590 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:22.366595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:22.366598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:22.366629 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:22.369088 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.369914 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.369924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.369926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.370094 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.370506 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.370515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.370517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.370777 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:22.370939 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.371612 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.371827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:55:22.373077 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.355811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37147 (local address 127.27.254.3:43233)
0808 15:55:22.356572 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.356615 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:22.358256 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:22.358979 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:22.358987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:22.359482 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:22.360010 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.360026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.361078 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.361082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.362429 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.362440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.363220 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.363227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.363433 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.364087 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:22.364107 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:22.366409 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:22.369216 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:22.369225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:22.369241 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:22.369785 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:22.370211 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:22.370216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:22.370219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:22.370383 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:22.370919 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:22.370926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:22.371237 (+ 311us) client_negotiation.cc:769] Sending connection context
0808 15:55:22.371486 (+ 249us) client_negotiation.cc:240] Negotiation successful
0808 15:55:22.371978 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
I20260808 15:55:22.374410 766 heartbeater.cc:347] Connected to a master server at 127.27.254.62:37147
I20260808 15:55:22.374694 766 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:22.375305 766 heartbeater.cc:511] Master 127.27.254.62:37147 requested a full tablet report, sending...
I20260808 15:55:22.376621 313 ts_manager.cc:194] Registered new tserver with Master: 81ace5453ad24984b449ad78f3c8b43f (127.27.254.3:37901)
I20260808 15:55:22.377390 313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:43233
I20260808 15:55:22.382751 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:55:22.382951 501 heartbeater.cc:503] Master 127.27.254.62:37147 was elected leader, sending a full tablet report...
I20260808 15:55:22.399855 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.387709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.27.254.62:37147)
0808 15:55:22.387936 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.387943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.388250 (+ 307us) server_negotiation.cc:407] Connection header received
0808 15:55:22.388382 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.388385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.388444 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.388519 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:22.389961 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.390651 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.391862 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.392277 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.392366 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.394546 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:22.394551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:22.394554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:22.394582 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:22.397110 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.397828 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.397832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.397834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.397974 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.398327 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.398330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.398332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.398656 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:22.398832 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.399258 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.399415 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:55:22.403682 313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:22.410496 313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:22.444676 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.435783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:41075 (local address 127.0.0.1:52990)
0808 15:55:22.436801 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.436819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:22.437022 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:22.437766 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:22.437771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:22.437805 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:22.438172 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.438179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.440046 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.440055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.442165 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.442176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.443714 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.443720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.444061 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.444090 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:55:22.444176 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 15:55:22.444281 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":177,"thread_start_us":695,"threads_started":1}
I20260808 15:55:22.444962 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.422429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.27.254.1:40041)
0808 15:55:22.435074 (+ 12645us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.435083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.435111 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:55:22.435225 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.435232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.435527 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.435728 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:22.436762 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.438360 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.441686 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.443167 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.443314 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.443557 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.444445 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.444620 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12463,"thread_start_us":12311,"threads_started":1}
I20260808 15:55:22.445480 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.435141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.27.254.2:41075)
0808 15:55:22.435812 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.435823 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.436912 (+ 1089us) server_negotiation.cc:407] Connection header received
0808 15:55:22.437206 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.437213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.437449 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.437624 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:22.438323 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.439883 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.442313 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.443404 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.443536 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.443727 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.444927 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.445195 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
I20260808 15:55:22.445767 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.422005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37901 (local address 127.0.0.1:46088)
0808 15:55:22.434389 (+ 12384us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.434404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:22.434499 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:22.437325 (+ 2826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:22.437329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:22.437352 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:22.437817 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.437832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.439367 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.439375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.442829 (+ 3454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.442844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.444884 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.444891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.445084 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.445113 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:55:22.445192 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 15:55:22.445312 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12269,"mutex_wait_us":205,"thread_start_us":11008,"threads_started":1}
I20260808 15:55:22.445837 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.422459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.27.254.3:37901)
0808 15:55:22.436516 (+ 14057us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.436525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.436558 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:55:22.436647 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.436654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.436895 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.437067 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:22.438059 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.439230 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.443005 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.444409 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.444533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.444762 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.445259 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.445426 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13828,"thread_start_us":13639,"threads_started":1}
I20260808 15:55:22.446468 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.422004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37004)
0808 15:55:22.434018 (+ 12014us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.434082 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:22.434327 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:22.435973 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:22.435983 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:22.436040 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:22.436533 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.436554 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.438525 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.438530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.441463 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.441478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.443372 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.443378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.443572 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.443919 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 15:55:22.444049 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 15:55:22.444245 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11383,"mutex_wait_us":20,"thread_start_us":12472,"threads_started":1}
I20260808 15:55:22.449574 569 tablet_service.cc:1467] Processing CreateTablet for tablet a3ee355b4e0746feb5ef9e8a748449f7 (DEFAULT_TABLE table=test-table [id=f43a4116fc43448cb0ad674313d7e277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:22.449684 436 tablet_service.cc:1467] Processing CreateTablet for tablet a3ee355b4e0746feb5ef9e8a748449f7 (DEFAULT_TABLE table=test-table [id=f43a4116fc43448cb0ad674313d7e277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:22.450789 569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ee355b4e0746feb5ef9e8a748449f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:22.450819 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ee355b4e0746feb5ef9e8a748449f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:22.455463 701 tablet_service.cc:1467] Processing CreateTablet for tablet a3ee355b4e0746feb5ef9e8a748449f7 (DEFAULT_TABLE table=test-table [id=f43a4116fc43448cb0ad674313d7e277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:22.456817 701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ee355b4e0746feb5ef9e8a748449f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:22.461946 784 tablet_bootstrap.cc:492] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Bootstrap starting.
I20260808 15:55:22.464434 785 tablet_bootstrap.cc:492] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Bootstrap starting.
I20260808 15:55:22.465497 784 tablet_bootstrap.cc:654] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:22.467020 784 log.cc:824] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:22.468115 785 tablet_bootstrap.cc:654] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:22.469650 785 log.cc:824] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:22.471434 788 tablet_bootstrap.cc:492] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Bootstrap starting.
I20260808 15:55:22.473057 785 tablet_bootstrap.cc:492] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: No bootstrap required, opened a new log
I20260808 15:55:22.473353 785 ts_tablet_manager.cc:1397] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:55:22.473883 784 tablet_bootstrap.cc:492] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: No bootstrap required, opened a new log
I20260808 15:55:22.474162 784 ts_tablet_manager.cc:1397] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260808 15:55:22.474922 788 tablet_bootstrap.cc:654] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:22.476058 788 log.cc:824] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:22.479032 788 tablet_bootstrap.cc:492] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: No bootstrap required, opened a new log
I20260808 15:55:22.479288 788 ts_tablet_manager.cc:1397] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:55:22.481842 784 raft_consensus.cc:348] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.482208 784 raft_consensus.cc:374] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:22.482326 784 raft_consensus.cc:729] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aec8f11262043f8bf3c9a4689b0bbac, State: Initialized, Role: FOLLOWER
I20260808 15:55:22.482955 784 consensus_queue.cc:260] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.483675 785 raft_consensus.cc:348] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.484078 785 raft_consensus.cc:374] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:22.484185 785 raft_consensus.cc:729] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85fae41340b141148781fa2a32630b1a, State: Initialized, Role: FOLLOWER
I20260808 15:55:22.484712 784 ts_tablet_manager.cc:1428] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260808 15:55:22.484902 785 consensus_queue.cc:260] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.488862 634 heartbeater.cc:503] Master 127.27.254.62:37147 was elected leader, sending a full tablet report...
I20260808 15:55:22.489674 785 ts_tablet_manager.cc:1428] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260808 15:55:22.492058 788 raft_consensus.cc:348] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.492445 788 raft_consensus.cc:374] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:22.492558 788 raft_consensus.cc:729] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ace5453ad24984b449ad78f3c8b43f, State: Initialized, Role: FOLLOWER
I20260808 15:55:22.493161 788 consensus_queue.cc:260] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.494153 766 heartbeater.cc:503] Master 127.27.254.62:37147 was elected leader, sending a full tablet report...
I20260808 15:55:22.494586 788 ts_tablet_manager.cc:1428] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260808 15:55:22.600628 502 tablet.cc:2369] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:55:22.604393 767 tablet.cc:2369] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:55:22.612365 635 tablet.cc:2369] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:22.664852 790 raft_consensus.cc:482] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:22.665148 790 raft_consensus.cc:504] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.666787 790 leader_election.cc:288] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81ace5453ad24984b449ad78f3c8b43f (127.27.254.3:37901), 85fae41340b141148781fa2a32630b1a (127.27.254.2:41075)
I20260808 15:55:22.673983 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.667506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:54309 (local address 127.27.254.2:41075)
0808 15:55:22.667819 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.667825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.668518 (+ 693us) server_negotiation.cc:407] Connection header received
0808 15:55:22.668711 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.668716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.668773 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.668872 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:22.669624 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.670329 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.672106 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.672906 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.673070 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.673143 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.673582 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.673730 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260808 15:55:22.674023 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.667256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:41075 (local address 127.27.254.1:54309)
0808 15:55:22.668412 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.668432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:22.668552 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:22.669056 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:22.669061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:22.669085 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:22.669458 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.669471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.670481 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.670486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.671947 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.671961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.673128 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.673135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.673331 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.673380 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 15:55:22.673521 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 15:55:22.673663 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":5,"thread_start_us":486,"threads_started":1}
I20260808 15:55:22.674141 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.667403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:42131 (local address 127.27.254.3:37901)
0808 15:55:22.667633 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.667642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.668262 (+ 620us) server_negotiation.cc:407] Connection header received
0808 15:55:22.668492 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.668497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.668581 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.668692 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:22.669214 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.670011 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.672007 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.672988 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.673157 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.673231 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.673766 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.673897 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:55:22.674830 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.667224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37901 (local address 127.27.254.1:42131)
0808 15:55:22.668115 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.668140 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:22.668329 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:22.668716 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:22.668721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:22.668747 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:22.669065 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.669074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.670156 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.670165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.671825 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:22.671845 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.673331 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.673338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:22.673578 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.673609 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:55:22.674305 (+ 696us) client_negotiation.cc:240] Negotiation successful
0808 15:55:22.674436 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":52,"thread_start_us":772,"threads_started":1}
I20260808 15:55:22.675010 589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ee355b4e0746feb5ef9e8a748449f7" candidate_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85fae41340b141148781fa2a32630b1a" is_pre_election: true
I20260808 15:55:22.675629 589 raft_consensus.cc:2454] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aec8f11262043f8bf3c9a4689b0bbac in term 0.
I20260808 15:55:22.675904 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ee355b4e0746feb5ef9e8a748449f7" candidate_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ace5453ad24984b449ad78f3c8b43f" is_pre_election: true
I20260808 15:55:22.676414 721 raft_consensus.cc:2454] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aec8f11262043f8bf3c9a4689b0bbac in term 0.
I20260808 15:55:22.676381 391 leader_election.cc:302] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aec8f11262043f8bf3c9a4689b0bbac, 85fae41340b141148781fa2a32630b1a; no voters:
I20260808 15:55:22.676823 790 raft_consensus.cc:2781] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:22.676980 790 raft_consensus.cc:482] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:22.677107 790 raft_consensus.cc:3037] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:22.679374 790 raft_consensus.cc:504] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.680009 790 leader_election.cc:288] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [CANDIDATE]: Term 1 election: Requested vote from peers 81ace5453ad24984b449ad78f3c8b43f (127.27.254.3:37901), 85fae41340b141148781fa2a32630b1a (127.27.254.2:41075)
I20260808 15:55:22.680335 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ee355b4e0746feb5ef9e8a748449f7" candidate_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ace5453ad24984b449ad78f3c8b43f"
I20260808 15:55:22.680532 721 raft_consensus.cc:3037] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:22.680476 589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ee355b4e0746feb5ef9e8a748449f7" candidate_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85fae41340b141148781fa2a32630b1a"
I20260808 15:55:22.680719 589 raft_consensus.cc:3037] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:22.682868 721 raft_consensus.cc:2454] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aec8f11262043f8bf3c9a4689b0bbac in term 1.
I20260808 15:55:22.682986 589 raft_consensus.cc:2454] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aec8f11262043f8bf3c9a4689b0bbac in term 1.
I20260808 15:55:22.683314 390 leader_election.cc:302] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aec8f11262043f8bf3c9a4689b0bbac, 81ace5453ad24984b449ad78f3c8b43f; no voters:
I20260808 15:55:22.683604 790 raft_consensus.cc:2781] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:22.684060 790 raft_consensus.cc:686] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 1 LEADER]: Becoming Leader. State: Replica: 3aec8f11262043f8bf3c9a4689b0bbac, State: Running, Role: LEADER
I20260808 15:55:22.684552 790 consensus_queue.cc:237] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } }
I20260808 15:55:22.689448 312 catalog_manager.cc:5495] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aec8f11262043f8bf3c9a4689b0bbac (127.27.254.1). New cstate: current_term: 1 leader_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aec8f11262043f8bf3c9a4689b0bbac" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:22.743687 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.738474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.27.254.1:40041)
0808 15:55:22.738693 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.738701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.738737 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:55:22.738991 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.738995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.739089 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.739248 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:22.739627 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.740426 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.741733 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.742154 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.742248 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.742652 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:22.743153 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:22.743266 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.743330 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.743392 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:55:22.751808 589 raft_consensus.cc:1261] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 1 FOLLOWER]: Refusing update from remote peer 3aec8f11262043f8bf3c9a4689b0bbac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:22.751807 721 raft_consensus.cc:1261] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 1 FOLLOWER]: Refusing update from remote peer 3aec8f11262043f8bf3c9a4689b0bbac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:22.752887 795 consensus_queue.cc:1035] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fae41340b141148781fa2a32630b1a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 41075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:22.753476 790 consensus_queue.cc:1035] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "81ace5453ad24984b449ad78f3c8b43f" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:22.766471 798 mvcc.cc:205] Tried to move back new op lower bound from 7316293725180588032 to 7316293724919894016. Current Snapshot: MvccSnapshot[applied={T|T < 7316293725180588032}]
I20260808 15:55:22.769520 800 mvcc.cc:205] Tried to move back new op lower bound from 7316293725180588032 to 7316293724919894016. Current Snapshot: MvccSnapshot[applied={T|T < 7316293725180588032}]
I20260808 15:55:22.777236 799 mvcc.cc:205] Tried to move back new op lower bound from 7316293725180588032 to 7316293724919894016. Current Snapshot: MvccSnapshot[applied={T|T < 7316293725180588032}]
I20260808 15:55:22.789352 313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:22.789716 313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:22.793800 313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24f264b039d54391b07d407ffac506e0: Sending DeleteTablet for 3 replicas of tablet a3ee355b4e0746feb5ef9e8a748449f7
I20260808 15:55:22.794900 701 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ee355b4e0746feb5ef9e8a748449f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:22 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:46088
I20260808 15:55:22.795439 436 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ee355b4e0746feb5ef9e8a748449f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:22 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37004
I20260808 15:55:22.795615 569 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ee355b4e0746feb5ef9e8a748449f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:22 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:52990
I20260808 15:55:22.796665 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 374
I20260808 15:55:22.818033 808 tablet_replica.cc:330] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: stopping tablet replica
I20260808 15:55:22.818207 810 tablet_replica.cc:330] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: stopping tablet replica
I20260808 15:55:22.818737 810 raft_consensus.cc:2229] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:22.818828 808 raft_consensus.cc:2229] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:22.819306 810 raft_consensus.cc:2258] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:22.821372 808 raft_consensus.cc:2258] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:22.821802 810 ts_tablet_manager.cc:1905] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:22.826457 809 tablet_replica.cc:330] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: stopping tablet replica
I20260808 15:55:22.826927 809 raft_consensus.cc:2229] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:22.827205 808 ts_tablet_manager.cc:1905] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:22.827611 809 raft_consensus.cc:2258] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 08 15:55:22 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:22.831472 810 ts_tablet_manager.cc:1918] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:22.831662 810 log.cc:1196] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/a3ee355b4e0746feb5ef9e8a748449f7
I20260808 15:55:22.832213 810 ts_tablet_manager.cc:1939] T a3ee355b4e0746feb5ef9e8a748449f7 P 85fae41340b141148781fa2a32630b1a: Deleting consensus metadata
I20260808 15:55:22.833613 808 ts_tablet_manager.cc:1918] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:22.833796 808 log.cc:1196] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/a3ee355b4e0746feb5ef9e8a748449f7
I20260808 15:55:22.834277 32767 catalog_manager.cc:4841] TS 85fae41340b141148781fa2a32630b1a (127.27.254.2:41075): tablet a3ee355b4e0746feb5ef9e8a748449f7 (table test-table [id=f43a4116fc43448cb0ad674313d7e277]) successfully deleted
I20260808 15:55:22.835083 808 ts_tablet_manager.cc:1939] T a3ee355b4e0746feb5ef9e8a748449f7 P 81ace5453ad24984b449ad78f3c8b43f: Deleting consensus metadata
I20260808 15:55:22.835903 809 ts_tablet_manager.cc:1905] T a3ee355b4e0746feb5ef9e8a748449f7 P 3aec8f11262043f8bf3c9a4689b0bbac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:22.836998 32766 catalog_manager.cc:4841] TS 81ace5453ad24984b449ad78f3c8b43f (127.27.254.3:37901): tablet a3ee355b4e0746feb5ef9e8a748449f7 (table test-table [id=f43a4116fc43448cb0ad674313d7e277]) successfully deleted
I20260808 15:55:22.840412 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.799334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.27.254.1:40041)
0808 15:55:22.799565 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.799575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.809787 (+ 10212us) server_negotiation.cc:407] Connection header received
0808 15:55:22.810471 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.810480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.810596 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.810711 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:22.813230 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.814981 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.819718 (+ 4737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.820743 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.820887 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.832285 (+ 11398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:22.832305 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:22.832325 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:22.832384 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:22.836998 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.838005 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.838012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.838014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.838191 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.838630 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.838634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.838636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.838990 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:22.839286 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.839619 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.839886 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:55:22.841215 496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:22.940122 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 374
W20260808 15:55:22.958093 32767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:40041 (error 108)
I20260808 15:55:22.958986 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 507
I20260808 15:55:22.960351 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.959495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37034)
0808 15:55:22.959996 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:22.960135 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":263,"threads_started":1}
W20260808 15:55:22.960642 32767 catalog_manager.cc:4584] TS 3aec8f11262043f8bf3c9a4689b0bbac (127.27.254.1:40041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3ee355b4e0746feb5ef9e8a748449f7: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Aug 08 15:55:22 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:55:22.973832 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:22.959834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.27.254.2:41075)
0808 15:55:22.960009 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:22.960014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:22.960190 (+ 176us) server_negotiation.cc:407] Connection header received
0808 15:55:22.960369 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:22.960373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:22.960447 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:22.960575 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:22.961887 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.962615 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.963802 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:22.964186 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:22.964276 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:22.968409 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:22.968422 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:22.968438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:22.968471 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:22.971261 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.971948 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.971955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.971958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.972079 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:22.972427 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:22.972431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:22.972433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:22.972786 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:22.972998 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:22.973309 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 15:55:22.973558 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:55:22.974563 629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:23.014498 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.014021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37044)
0808 15:55:23.014245 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:23.014340 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 15:55:23.015654 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.015298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37046)
0808 15:55:23.015447 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:23.015522 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 15:55:23.063702 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 507
I20260808 15:55:23.075978 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.075523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37050)
0808 15:55:23.075748 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:23.075840 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 15:55:23.076998 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.076634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37060)
0808 15:55:23.076791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:23.076867 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:55:23.080829 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 639
Aug 08 15:55:23 dist-test-slave-srgv krb5kdc[32737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:55:23.095353 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.081596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.27.254.3:37901)
0808 15:55:23.081760 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:23.081764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:23.081783 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:55:23.081946 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:23.081950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:23.082014 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:23.082124 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:23.083364 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:23.084061 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:23.085337 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:23.085824 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:23.085921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:23.090232 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:23.090247 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:23.090263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:23.090299 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:23.092824 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:23.093533 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:23.093545 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:23.093549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:23.093696 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:23.094037 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:23.094040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:23.094043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:23.094400 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:23.094602 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:23.094930 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 15:55:23.095102 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:55:23.096032 761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:23.158355 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.157842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37066)
0808 15:55:23.158096 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:23.158195 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 15:55:23.159525 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.159091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40041 (local address 127.0.0.1:37070)
0808 15:55:23.159328 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:23.159397 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 15:55:23.185722 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 639
I20260808 15:55:23.205536 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 32750
I20260808 15:55:23.206717 343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:23.313766 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 32750
2026-08-08T15:55:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4254 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T15:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:23Z Disabled control of system clock
I20260808 15:55:23.379557 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40899
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 15:55:23.701743 821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:23.702029 821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:23.702114 821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:23.709296 821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:23.709460 821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:23.709527 821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:23.709594 821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:23.709640 821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:23.719131 821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:40899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:23.720378 821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:23.722162 821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:23.734119 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:23.734170 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:23.734364 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:23.735139 821 server_base.cc:1034] running on GCE node
I20260808 15:55:23.735925 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:23.737341 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:23.738601 821 hybrid_clock.cc:648] HybridClock initialized: now 1786204523738553 us; error 41 us; skew 500 ppm
I20260808 15:55:23.738969 821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:23.741242 821 webserver.cc:460] Webserver started at http://127.27.254.62:41601/ using document root <none> and password file <none>
I20260808 15:55:23.741994 821 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:23.742142 821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:23.742463 821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:23.745364 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "94978c9586e34cde88766808463ae112"
format_stamp: "Formatted at 2026-08-08 15:55:23 on dist-test-slave-srgv"
server_key: "f2fdbe3baaa618fa594ccd3627e7690c"
server_key_iv: "83f10ef30288b999c5576ebcdffd95f8"
server_key_version: "encryptionkey@0"
I20260808 15:55:23.746222 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "94978c9586e34cde88766808463ae112"
format_stamp: "Formatted at 2026-08-08 15:55:23 on dist-test-slave-srgv"
server_key: "f2fdbe3baaa618fa594ccd3627e7690c"
server_key_iv: "83f10ef30288b999c5576ebcdffd95f8"
server_key_version: "encryptionkey@0"
I20260808 15:55:23.751263 821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:23.754736 835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:23.755968 821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:55:23.756173 821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "94978c9586e34cde88766808463ae112"
format_stamp: "Formatted at 2026-08-08 15:55:23 on dist-test-slave-srgv"
server_key: "f2fdbe3baaa618fa594ccd3627e7690c"
server_key_iv: "83f10ef30288b999c5576ebcdffd95f8"
server_key_version: "encryptionkey@0"
I20260808 15:55:23.756397 821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:23.762629 821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:23.763404 821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:23.763711 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:23.782130 821 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:40899
I20260808 15:55:23.782151 886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:40899 every 8 connection(s)
I20260808 15:55:23.783499 821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:23.787273 887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:23.787982 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 821
I20260808 15:55:23.788208 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:23.789114 28664 external_mini_cluster.cc:1428] Setting key d8d79411808c32d07366e71c0dcd4326
I20260808 15:55:23.797923 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:23.791599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.27.254.62:40899)
0808 15:55:23.792717 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:23.792736 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:23.792781 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:55:23.794367 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:23.794392 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:23.794837 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:23.795186 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:23.796251 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:23.796257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:23.796269 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:23.796283 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:23.796345 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:23.796361 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:23.796533 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:23.796654 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 15:55:23.796900 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":450,"threads_started":1}
I20260808 15:55:23.801049 887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112: Bootstrap starting.
I20260808 15:55:23.804075 887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:23.805007 887 log.cc:824] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:23.807655 887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112: No bootstrap required, opened a new log
I20260808 15:55:23.814585 887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94978c9586e34cde88766808463ae112" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40899 } }
I20260808 15:55:23.814962 887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:23.815064 887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94978c9586e34cde88766808463ae112, State: Initialized, Role: FOLLOWER
I20260808 15:55:23.815747 887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94978c9586e34cde88766808463ae112" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40899 } }
I20260808 15:55:23.815991 887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:23.816126 887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:23.816305 887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:23.818478 887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94978c9586e34cde88766808463ae112" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40899 } }
I20260808 15:55:23.818984 887 leader_election.cc:302] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94978c9586e34cde88766808463ae112; no voters:
I20260808 15:55:23.819646 887 leader_election.cc:288] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:23.819788 892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:23.820494 892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [term 1 LEADER]: Becoming Leader. State: Replica: 94978c9586e34cde88766808463ae112, State: Running, Role: LEADER
I20260808 15:55:23.821626 892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94978c9586e34cde88766808463ae112" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40899 } }
I20260808 15:55:23.822294 887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:23.824609 894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94978c9586e34cde88766808463ae112. Latest consensus state: current_term: 1 leader_uuid: "94978c9586e34cde88766808463ae112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94978c9586e34cde88766808463ae112" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40899 } } }
I20260808 15:55:23.824982 894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:23.824795 893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94978c9586e34cde88766808463ae112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94978c9586e34cde88766808463ae112" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40899 } } }
I20260808 15:55:23.825220 893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:23.826782 900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:23.831749 900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:23.839545 900 catalog_manager.cc:1347] Generated new cluster ID: 8d5f69b00667422dae3ac518723c96d3
I20260808 15:55:23.839664 900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:23.873473 900 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:23.873948 900 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:23.887225 900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112: Generated new TSK 0
I20260808 15:55:23.888258 900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:55:23.896795 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40899
--builtin_ntp_servers=127.27.254.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 15:55:24.226096 911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:24.226420 911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:24.226606 911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:24.234066 911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:24.234194 911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:24.234381 911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:24.243141 911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:24.244411 911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:24.246246 911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:24.263428 916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:24.263514 919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:24.263481 917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:24.264472 911 server_base.cc:1034] running on GCE node
I20260808 15:55:24.265075 911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:24.266213 911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:24.267412 911 hybrid_clock.cc:648] HybridClock initialized: now 1786204524267369 us; error 44 us; skew 500 ppm
I20260808 15:55:24.267740 911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:24.269935 911 webserver.cc:460] Webserver started at http://127.27.254.1:37663/ using document root <none> and password file <none>
I20260808 15:55:24.270637 911 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:24.270762 911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:24.271090 911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:24.273983 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "b7285034ef7d4ed2a0a7d08847e8505e"
format_stamp: "Formatted at 2026-08-08 15:55:24 on dist-test-slave-srgv"
server_key: "6927c5c96991c366aee096be15db3d03"
server_key_iv: "26451ac0c26086465b4973959d8fa82e"
server_key_version: "encryptionkey@0"
I20260808 15:55:24.274744 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "b7285034ef7d4ed2a0a7d08847e8505e"
format_stamp: "Formatted at 2026-08-08 15:55:24 on dist-test-slave-srgv"
server_key: "6927c5c96991c366aee096be15db3d03"
server_key_iv: "26451ac0c26086465b4973959d8fa82e"
server_key_version: "encryptionkey@0"
I20260808 15:55:24.279868 911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:55:24.283221 925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:24.284472 911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:55:24.284641 911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "b7285034ef7d4ed2a0a7d08847e8505e"
format_stamp: "Formatted at 2026-08-08 15:55:24 on dist-test-slave-srgv"
server_key: "6927c5c96991c366aee096be15db3d03"
server_key_iv: "26451ac0c26086465b4973959d8fa82e"
server_key_version: "encryptionkey@0"
I20260808 15:55:24.284855 911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:24.290403 911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:24.291146 911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:24.291414 911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:24.292248 911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:24.293646 911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:24.293763 911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:24.293889 911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:24.293988 911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:24.326792 911 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:34601
I20260808 15:55:24.326958 1037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:34601 every 8 connection(s)
I20260808 15:55:24.328210 911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:24.336544 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:24.330875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:48651 (local address 127.27.254.62:40899)
0808 15:55:24.331343 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:24.331350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:24.332634 (+ 1284us) server_negotiation.cc:407] Connection header received
0808 15:55:24.334412 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:24.334416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:24.334466 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:24.334563 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:24.335953 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:24.335956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:24.335959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:24.335972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:24.336007 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:24.336018 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:24.336112 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:24.336289 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 15:55:24.336337 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260808 15:55:24.337442 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 911
I20260808 15:55:24.337591 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:24.331498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40899 (local address 127.27.254.1:48651)
0808 15:55:24.332426 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:24.332495 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:24.334176 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:24.334817 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:24.334826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:24.334919 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:24.335664 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:24.335666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:24.335705 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:24.335789 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:24.336113 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:24.336120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:24.336126 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 15:55:24.336340 (+ 214us) client_negotiation.cc:240] Negotiation successful
0808 15:55:24.336537 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260808 15:55:24.337637 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:24.338430 28664 external_mini_cluster.cc:1428] Setting key 430defe343bbe94c84cabc943ff11729
I20260808 15:55:24.339205 1038 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40899
I20260808 15:55:24.339540 1038 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:24.340246 1038 heartbeater.cc:511] Master 127.27.254.62:40899 requested a full tablet report, sending...
I20260808 15:55:24.341600 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40899
--builtin_ntp_servers=127.27.254.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 15:55:24.342427 852 ts_manager.cc:194] Registered new tserver with Master: b7285034ef7d4ed2a0a7d08847e8505e (127.27.254.1:34601)
W20260808 15:55:24.659404 1042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:24.659710 1042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:24.659878 1042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:24.666961 1042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:24.667092 1042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:24.667299 1042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:24.676231 1042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:24.677557 1042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:24.679327 1042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:24.692062 1047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:24.692030 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:24.692394 1050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:24.692632 1042 server_base.cc:1034] running on GCE node
I20260808 15:55:24.693362 1042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:24.694278 1042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:24.695508 1042 hybrid_clock.cc:648] HybridClock initialized: now 1786204524695466 us; error 58 us; skew 500 ppm
I20260808 15:55:24.695828 1042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:24.698117 1042 webserver.cc:460] Webserver started at http://127.27.254.2:42403/ using document root <none> and password file <none>
I20260808 15:55:24.698809 1042 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:24.698942 1042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:24.699283 1042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:24.702167 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "30b1d552643c498aafa9d934d5b7c3d3"
format_stamp: "Formatted at 2026-08-08 15:55:24 on dist-test-slave-srgv"
server_key: "4d6fd70ae9344d68ff1502edc23e3ab3"
server_key_iv: "5b623d6a9cdedbe76e3043b6d9400590"
server_key_version: "encryptionkey@0"
I20260808 15:55:24.702941 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "30b1d552643c498aafa9d934d5b7c3d3"
format_stamp: "Formatted at 2026-08-08 15:55:24 on dist-test-slave-srgv"
server_key: "4d6fd70ae9344d68ff1502edc23e3ab3"
server_key_iv: "5b623d6a9cdedbe76e3043b6d9400590"
server_key_version: "encryptionkey@0"
I20260808 15:55:24.708082 1042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:55:24.711635 1056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:24.713008 1042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:24.713193 1042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "30b1d552643c498aafa9d934d5b7c3d3"
format_stamp: "Formatted at 2026-08-08 15:55:24 on dist-test-slave-srgv"
server_key: "4d6fd70ae9344d68ff1502edc23e3ab3"
server_key_iv: "5b623d6a9cdedbe76e3043b6d9400590"
server_key_version: "encryptionkey@0"
I20260808 15:55:24.713415 1042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:24.719066 1042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:24.719861 1042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:24.720144 1042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:24.721042 1042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:24.722515 1042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:24.722633 1042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:24.722754 1042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:24.722836 1042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:24.754935 1042 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:40513
I20260808 15:55:24.755069 1168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:40513 every 8 connection(s)
I20260808 15:55:24.756283 1042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:24.759902 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1042
I20260808 15:55:24.760103 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:24.760679 28664 external_mini_cluster.cc:1428] Setting key 6745fd20c31e6742d53f28c7e8141099
I20260808 15:55:24.763341 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40899
--builtin_ntp_servers=127.27.254.20:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 15:55:24.765161 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:24.758835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:51883 (local address 127.27.254.62:40899)
0808 15:55:24.759025 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:24.759030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:24.760820 (+ 1790us) server_negotiation.cc:407] Connection header received
0808 15:55:24.762854 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:24.762858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:24.762921 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:24.763039 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:24.764411 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:24.764417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:24.764421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:24.764439 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:24.764478 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:24.764489 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:24.764604 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:24.764818 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 15:55:24.764886 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:55:24.766237 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:24.759401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40899 (local address 127.27.254.2:51883)
0808 15:55:24.760641 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:24.760683 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:24.762589 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:24.763269 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:24.763296 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:24.763391 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:24.764129 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:24.764131 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:24.764170 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:24.764254 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:24.764579 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:24.764589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:24.764596 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:55:24.764901 (+ 305us) client_negotiation.cc:240] Negotiation successful
0808 15:55:24.765156 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":343,"threads_started":1}
I20260808 15:55:24.767570 1169 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40899
I20260808 15:55:24.767859 1169 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:24.768437 1169 heartbeater.cc:511] Master 127.27.254.62:40899 requested a full tablet report, sending...
I20260808 15:55:24.769936 852 ts_manager.cc:194] Registered new tserver with Master: 30b1d552643c498aafa9d934d5b7c3d3 (127.27.254.2:40513)
W20260808 15:55:25.092625 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:25.092896 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:25.093075 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:25.100219 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:25.100311 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:25.100522 1172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:25.109315 1172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:25.110580 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:25.112352 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:25.125025 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:25.125084 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:25.125486 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:25.125947 1172 server_base.cc:1034] running on GCE node
I20260808 15:55:25.126582 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:25.127562 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:25.128793 1172 hybrid_clock.cc:648] HybridClock initialized: now 1786204525128729 us; error 54 us; skew 500 ppm
I20260808 15:55:25.129099 1172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:25.131665 1172 webserver.cc:460] Webserver started at http://127.27.254.3:44479/ using document root <none> and password file <none>
I20260808 15:55:25.132360 1172 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:25.132484 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:25.132818 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:25.135648 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "78f713b7013a4ba9b6620614cf638ba1"
format_stamp: "Formatted at 2026-08-08 15:55:25 on dist-test-slave-srgv"
server_key: "3fbde734b218bdb4abe34cf1f27bbef7"
server_key_iv: "f66f3906275a6f7ed691c399292b3740"
server_key_version: "encryptionkey@0"
I20260808 15:55:25.136409 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "78f713b7013a4ba9b6620614cf638ba1"
format_stamp: "Formatted at 2026-08-08 15:55:25 on dist-test-slave-srgv"
server_key: "3fbde734b218bdb4abe34cf1f27bbef7"
server_key_iv: "f66f3906275a6f7ed691c399292b3740"
server_key_version: "encryptionkey@0"
I20260808 15:55:25.141528 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 15:55:25.144774 1186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:25.146090 1172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:55:25.146255 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "78f713b7013a4ba9b6620614cf638ba1"
format_stamp: "Formatted at 2026-08-08 15:55:25 on dist-test-slave-srgv"
server_key: "3fbde734b218bdb4abe34cf1f27bbef7"
server_key_iv: "f66f3906275a6f7ed691c399292b3740"
server_key_version: "encryptionkey@0"
I20260808 15:55:25.146453 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:25.152016 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:25.152741 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:25.152998 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:25.153959 1172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:25.155239 1172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:25.155364 1172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:25.155493 1172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:25.155578 1172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:25.188783 1172 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:45533
I20260808 15:55:25.188827 1298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:45533 every 8 connection(s)
I20260808 15:55:25.190200 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:25.193715 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1172
I20260808 15:55:25.193892 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:25.194446 28664 external_mini_cluster.cc:1428] Setting key 1597cd1e9832979e81c966dbd85194dd
I20260808 15:55:25.198827 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.192606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:49861 (local address 127.27.254.62:40899)
0808 15:55:25.192809 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.192815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.194463 (+ 1648us) server_negotiation.cc:407] Connection header received
0808 15:55:25.196479 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.196486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.196548 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.196627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.198066 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.198072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.198076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.198094 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.198132 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.198152 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.198269 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.198466 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.198533 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:55:25.199695 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.193171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40899 (local address 127.27.254.3:49861)
0808 15:55:25.194293 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.194338 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:25.196243 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:25.196891 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:25.196902 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:25.197023 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:25.197796 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:25.197798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:25.197835 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:25.197911 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:25.198245 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:25.198255 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:25.198262 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:55:25.198547 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 15:55:25.198820 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":349,"threads_started":1}
I20260808 15:55:25.200857 1299 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40899
I20260808 15:55:25.201144 1299 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:25.201753 1299 heartbeater.cc:511] Master 127.27.254.62:40899 requested a full tablet report, sending...
I20260808 15:55:25.203001 852 ts_manager.cc:194] Registered new tserver with Master: 78f713b7013a4ba9b6620614cf638ba1 (127.27.254.3:45533)
I20260808 15:55:25.210026 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:55:25.216723 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.214361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.27.254.62:40899)
0808 15:55:25.214526 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.214530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.215098 (+ 568us) server_negotiation.cc:407] Connection header received
0808 15:55:25.215232 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.215235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.215284 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.215362 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.216255 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.216258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.216261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.216274 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.216299 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.216309 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.216397 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.216440 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.216480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:55:25.220726 852 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:25.228125 852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:25.256906 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.240478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.27.254.3:45533)
0808 15:55:25.252221 (+ 11743us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.252232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.253299 (+ 1067us) server_negotiation.cc:407] Connection header received
0808 15:55:25.253392 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.253401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.253682 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.254512 (+ 830us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.255781 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.255793 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.255808 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.255829 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.255883 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.255905 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.256084 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.256371 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.256475 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11534,"thread_start_us":11369,"threads_started":1}
I20260808 15:55:25.256959 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.240294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:45533 (local address 127.0.0.1:38216)
0808 15:55:25.252756 (+ 12462us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.252811 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:25.253020 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:25.253926 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:25.253934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:25.254027 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:25.254839 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:25.254845 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:25.254881 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:25.254980 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:25.256141 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:25.256149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:25.256157 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:55:25.256372 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 15:55:25.256465 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12111,"thread_start_us":12925,"threads_started":1}
I20260808 15:55:25.257510 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.240586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.27.254.2:40513)
0808 15:55:25.252903 (+ 12317us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.252912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.253666 (+ 754us) server_negotiation.cc:407] Connection header received
0808 15:55:25.253919 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.253927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.254201 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.254335 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.256362 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.256370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.256383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.256402 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.256462 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.256488 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.256639 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.257063 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.257175 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12145,"thread_start_us":12305,"threads_started":1}
I20260808 15:55:25.257886 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.240328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40513 (local address 127.0.0.1:44100)
0808 15:55:25.253539 (+ 13211us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.253556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:25.253758 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:25.255419 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:25.255426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:25.255451 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:25.256114 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:25.256118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:25.256151 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:25.256195 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:25.256699 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:25.256704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:25.256714 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:55:25.256867 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 15:55:25.256929 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13055,"spinlock_wait_cycles":1920,"thread_start_us":11878,"threads_started":1}
I20260808 15:55:25.260169 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.254792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39880)
0808 15:55:25.257181 (+ 2389us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.257197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:25.257284 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:25.257979 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:25.257984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:25.258028 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:25.259230 (+ 1202us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:25.259234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:25.259249 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:25.259293 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:25.259764 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:25.259768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:25.259773 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:55:25.259865 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:55:25.259911 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2292,"thread_start_us":1239,"threads_started":1}
I20260808 15:55:25.260739 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.253926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.27.254.1:34601)
0808 15:55:25.254461 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.254476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.257385 (+ 2909us) server_negotiation.cc:407] Connection header received
0808 15:55:25.257487 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.257497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.257700 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.258469 (+ 769us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.259459 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.259467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.259479 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.259499 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.259556 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.259574 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.259710 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.260382 (+ 672us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.260475 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":251,"threads_started":1}
I20260808 15:55:25.261261 1234 tablet_service.cc:1467] Processing CreateTablet for tablet 840de5677a664137a69463f264c3eb50 (DEFAULT_TABLE table=test-table [id=87de43d4762e49d185e157085ced6a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:25.261757 1104 tablet_service.cc:1467] Processing CreateTablet for tablet 840de5677a664137a69463f264c3eb50 (DEFAULT_TABLE table=test-table [id=87de43d4762e49d185e157085ced6a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:25.262665 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840de5677a664137a69463f264c3eb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:25.262959 1104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840de5677a664137a69463f264c3eb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:25.268034 973 tablet_service.cc:1467] Processing CreateTablet for tablet 840de5677a664137a69463f264c3eb50 (DEFAULT_TABLE table=test-table [id=87de43d4762e49d185e157085ced6a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:25.269503 973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840de5677a664137a69463f264c3eb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:25.274487 1317 tablet_bootstrap.cc:492] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Bootstrap starting.
I20260808 15:55:25.278496 1317 tablet_bootstrap.cc:654] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:25.279511 1317 log.cc:824] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:25.280462 1318 tablet_bootstrap.cc:492] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Bootstrap starting.
I20260808 15:55:25.284060 1318 tablet_bootstrap.cc:654] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:25.284533 1317 tablet_bootstrap.cc:492] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: No bootstrap required, opened a new log
I20260808 15:55:25.284709 1320 tablet_bootstrap.cc:492] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Bootstrap starting.
I20260808 15:55:25.284893 1317 ts_tablet_manager.cc:1397] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 15:55:25.285378 1318 log.cc:824] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:25.287957 1320 tablet_bootstrap.cc:654] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:25.288486 1318 tablet_bootstrap.cc:492] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: No bootstrap required, opened a new log
I20260808 15:55:25.288825 1318 ts_tablet_manager.cc:1397] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:55:25.289120 1320 log.cc:824] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:25.291898 1320 tablet_bootstrap.cc:492] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: No bootstrap required, opened a new log
I20260808 15:55:25.292155 1320 ts_tablet_manager.cc:1397] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 15:55:25.295205 1317 raft_consensus.cc:348] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.295644 1317 raft_consensus.cc:374] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:25.295753 1317 raft_consensus.cc:729] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f713b7013a4ba9b6620614cf638ba1, State: Initialized, Role: FOLLOWER
I20260808 15:55:25.296368 1317 consensus_queue.cc:260] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.297490 1299 heartbeater.cc:503] Master 127.27.254.62:40899 was elected leader, sending a full tablet report...
I20260808 15:55:25.298174 1317 ts_tablet_manager.cc:1428] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260808 15:55:25.299191 1318 raft_consensus.cc:348] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.299535 1318 raft_consensus.cc:374] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:25.299659 1318 raft_consensus.cc:729] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b1d552643c498aafa9d934d5b7c3d3, State: Initialized, Role: FOLLOWER
I20260808 15:55:25.300237 1318 consensus_queue.cc:260] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.300172 1320 raft_consensus.cc:348] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.300593 1320 raft_consensus.cc:374] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:25.300702 1320 raft_consensus.cc:729] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7285034ef7d4ed2a0a7d08847e8505e, State: Initialized, Role: FOLLOWER
I20260808 15:55:25.301386 1169 heartbeater.cc:503] Master 127.27.254.62:40899 was elected leader, sending a full tablet report...
I20260808 15:55:25.301263 1320 consensus_queue.cc:260] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.303438 1318 ts_tablet_manager.cc:1428] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260808 15:55:25.304298 1038 heartbeater.cc:503] Master 127.27.254.62:40899 was elected leader, sending a full tablet report...
I20260808 15:55:25.304977 1320 ts_tablet_manager.cc:1428] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260808 15:55:25.330579 1039 tablet.cc:2369] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:25.383021 1323 raft_consensus.cc:482] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:25.383314 1323 raft_consensus.cc:504] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.384887 1323 leader_election.cc:288] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30b1d552643c498aafa9d934d5b7c3d3 (127.27.254.2:40513), b7285034ef7d4ed2a0a7d08847e8505e (127.27.254.1:34601)
I20260808 15:55:25.387583 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.385331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40513 (local address 127.27.254.3:42509)
0808 15:55:25.385526 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.385544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:25.385669 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:25.385981 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:25.385988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:25.386019 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:25.386754 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:25.386758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:25.386775 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:25.386825 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:25.387153 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:25.387157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:25.387160 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 15:55:25.387237 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 15:55:25.387311 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 15:55:25.387579 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.385182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:42509 (local address 127.27.254.2:40513)
0808 15:55:25.385426 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.385433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.385628 (+ 195us) server_negotiation.cc:407] Connection header received
0808 15:55:25.385782 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.385785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.385833 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.385914 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.386958 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.386964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.386968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.386989 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.387031 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.387050 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.387193 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.387246 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.387310 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:55:25.388559 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.385734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.27.254.3:39021)
0808 15:55:25.386469 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.386486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:25.386591 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:25.386930 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:25.386934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:25.386955 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:25.387602 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:55:25.387607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:25.387623 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:55:25.387675 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:25.388087 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:25.388090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:25.388093 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 15:55:25.388173 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 15:55:25.388209 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":402,"threads_started":1}
I20260808 15:55:25.388607 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.385361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:39021 (local address 127.27.254.1:34601)
0808 15:55:25.385548 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.385551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.386591 (+ 1040us) server_negotiation.cc:407] Connection header received
0808 15:55:25.386710 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.386716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.386776 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.386880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.387841 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.387847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.387854 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.387894 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.387929 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.387946 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.388020 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.388218 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.388277 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 15:55:25.388542 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840de5677a664137a69463f264c3eb50" candidate_uuid: "78f713b7013a4ba9b6620614cf638ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1d552643c498aafa9d934d5b7c3d3" is_pre_election: true
I20260808 15:55:25.389108 1124 raft_consensus.cc:2454] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f713b7013a4ba9b6620614cf638ba1 in term 0.
I20260808 15:55:25.389549 993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840de5677a664137a69463f264c3eb50" candidate_uuid: "78f713b7013a4ba9b6620614cf638ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" is_pre_election: true
I20260808 15:55:25.389863 1187 leader_election.cc:302] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b1d552643c498aafa9d934d5b7c3d3, 78f713b7013a4ba9b6620614cf638ba1; no voters:
I20260808 15:55:25.390117 993 raft_consensus.cc:2454] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f713b7013a4ba9b6620614cf638ba1 in term 0.
I20260808 15:55:25.390277 1323 raft_consensus.cc:2781] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:25.390429 1323 raft_consensus.cc:482] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:25.390532 1323 raft_consensus.cc:3037] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:25.392668 1323 raft_consensus.cc:504] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.393201 1323 leader_election.cc:288] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [CANDIDATE]: Term 1 election: Requested vote from peers 30b1d552643c498aafa9d934d5b7c3d3 (127.27.254.2:40513), b7285034ef7d4ed2a0a7d08847e8505e (127.27.254.1:34601)
I20260808 15:55:25.393640 993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840de5677a664137a69463f264c3eb50" candidate_uuid: "78f713b7013a4ba9b6620614cf638ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7285034ef7d4ed2a0a7d08847e8505e"
I20260808 15:55:25.393674 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840de5677a664137a69463f264c3eb50" candidate_uuid: "78f713b7013a4ba9b6620614cf638ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1d552643c498aafa9d934d5b7c3d3"
I20260808 15:55:25.393811 993 raft_consensus.cc:3037] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:25.393885 1124 raft_consensus.cc:3037] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:25.395916 993 raft_consensus.cc:2454] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f713b7013a4ba9b6620614cf638ba1 in term 1.
I20260808 15:55:25.396018 1124 raft_consensus.cc:2454] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f713b7013a4ba9b6620614cf638ba1 in term 1.
I20260808 15:55:25.396310 1190 leader_election.cc:302] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f713b7013a4ba9b6620614cf638ba1, b7285034ef7d4ed2a0a7d08847e8505e; no voters:
I20260808 15:55:25.396621 1323 raft_consensus.cc:2781] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:25.397043 1323 raft_consensus.cc:686] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 78f713b7013a4ba9b6620614cf638ba1, State: Running, Role: LEADER
I20260808 15:55:25.397543 1323 consensus_queue.cc:237] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } }
I20260808 15:55:25.402230 851 catalog_manager.cc:5495] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78f713b7013a4ba9b6620614cf638ba1 (127.27.254.3). New cstate: current_term: 1 leader_uuid: "78f713b7013a4ba9b6620614cf638ba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f713b7013a4ba9b6620614cf638ba1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 } health_report { overall_health: UNKNOWN } } }
W20260808 15:55:25.441972 1300 tablet.cc:2369] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:25.451152 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.448988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.27.254.3:45533)
0808 15:55:25.449192 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.449197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.449217 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:55:25.449466 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.449471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.449547 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.449662 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.450613 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.450617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.450620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.450639 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.450666 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.450676 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.450814 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.450849 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.450892 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:55:25.459352 993 raft_consensus.cc:1261] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 1 FOLLOWER]: Refusing update from remote peer 78f713b7013a4ba9b6620614cf638ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:25.459362 1124 raft_consensus.cc:1261] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 1 FOLLOWER]: Refusing update from remote peer 78f713b7013a4ba9b6620614cf638ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:25.460438 1327 consensus_queue.cc:1035] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7285034ef7d4ed2a0a7d08847e8505e" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:25.460819 1323 consensus_queue.cc:1035] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30b1d552643c498aafa9d934d5b7c3d3" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:25.473048 1331 mvcc.cc:205] Tried to move back new op lower bound from 7316293736271892480 to 7316293736032038912. Current Snapshot: MvccSnapshot[applied={T|T < 7316293736271892480}]
I20260808 15:55:25.473440 1332 mvcc.cc:205] Tried to move back new op lower bound from 7316293736271892480 to 7316293736032038912. Current Snapshot: MvccSnapshot[applied={T|T < 7316293736271892480}]
I20260808 15:55:25.487758 851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:25.488075 851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:25.491825 851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94978c9586e34cde88766808463ae112: Sending DeleteTablet for 3 replicas of tablet 840de5677a664137a69463f264c3eb50
I20260808 15:55:25.493003 1104 tablet_service.cc:1514] Processing DeleteTablet for tablet 840de5677a664137a69463f264c3eb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:25 UTC) from {username='slave'} at 127.0.0.1:44100
I20260808 15:55:25.493063 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet 840de5677a664137a69463f264c3eb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:25 UTC) from {username='slave'} at 127.0.0.1:38216
I20260808 15:55:25.493969 973 tablet_service.cc:1514] Processing DeleteTablet for tablet 840de5677a664137a69463f264c3eb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:25 UTC) from {username='slave'} at 127.0.0.1:39880
I20260808 15:55:25.494133 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 911
I20260808 15:55:25.498553 1342 tablet_replica.cc:330] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: stopping tablet replica
I20260808 15:55:25.499159 1342 raft_consensus.cc:2229] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:25.499833 1342 raft_consensus.cc:2258] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:25.500692 1341 tablet_replica.cc:330] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: stopping tablet replica
I20260808 15:55:25.501155 1341 raft_consensus.cc:2229] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:25.501910 1341 raft_consensus.cc:2258] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:25.502952 1342 ts_tablet_manager.cc:1905] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:25.503772 1341 ts_tablet_manager.cc:1905] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:25.506387 1340 tablet_replica.cc:330] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: stopping tablet replica
I20260808 15:55:25.506855 1340 raft_consensus.cc:2229] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:25.507412 1340 raft_consensus.cc:2258] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:25.509529 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.495379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.27.254.1:34601)
0808 15:55:25.495550 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.495556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.497903 (+ 2347us) server_negotiation.cc:407] Connection header received
0808 15:55:25.498090 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.498099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.498238 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.498359 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.508622 (+ 10263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.508628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.508632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.508654 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.508691 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.508707 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.508831 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.509097 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.509165 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:55:25.510542 1033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:25.511509 1341 ts_tablet_manager.cc:1918] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:25.511621 1340 ts_tablet_manager.cc:1905] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:25.511861 1341 log.cc:1196] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/840de5677a664137a69463f264c3eb50
I20260808 15:55:25.512622 1341 ts_tablet_manager.cc:1939] T 840de5677a664137a69463f264c3eb50 P 78f713b7013a4ba9b6620614cf638ba1: Deleting consensus metadata
I20260808 15:55:25.514955 839 catalog_manager.cc:4841] TS 78f713b7013a4ba9b6620614cf638ba1 (127.27.254.3:45533): tablet 840de5677a664137a69463f264c3eb50 (table test-table [id=87de43d4762e49d185e157085ced6a67]) successfully deleted
I20260808 15:55:25.518472 1340 ts_tablet_manager.cc:1918] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:25.518646 1340 log.cc:1196] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/840de5677a664137a69463f264c3eb50
I20260808 15:55:25.519197 1340 ts_tablet_manager.cc:1939] T 840de5677a664137a69463f264c3eb50 P 30b1d552643c498aafa9d934d5b7c3d3: Deleting consensus metadata
I20260808 15:55:25.521075 836 catalog_manager.cc:4841] TS 30b1d552643c498aafa9d934d5b7c3d3 (127.27.254.2:40513): tablet 840de5677a664137a69463f264c3eb50 (table test-table [id=87de43d4762e49d185e157085ced6a67]) successfully deleted
I20260808 15:55:25.628872 1342 ts_tablet_manager.cc:1918] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:25.629142 1342 log.cc:1196] T 840de5677a664137a69463f264c3eb50 P b7285034ef7d4ed2a0a7d08847e8505e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/840de5677a664137a69463f264c3eb50
I20260808 15:55:25.629570 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 911
W20260808 15:55:25.646236 839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:34601 (error 108)
I20260808 15:55:25.647033 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1042
I20260808 15:55:25.648465 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.647893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39894)
0808 15:55:25.648159 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.648285 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260808 15:55:25.648751 839 catalog_manager.cc:4584] TS b7285034ef7d4ed2a0a7d08847e8505e (127.27.254.1:34601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 840de5677a664137a69463f264c3eb50: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
I20260808 15:55:25.650516 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.648000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.27.254.2:40513)
0808 15:55:25.648179 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.648185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.648206 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:55:25.648399 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.648405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.648510 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.648644 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.649738 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.649746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.649752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.649771 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.649806 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.649827 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.649962 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.650125 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.650191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:55:25.651264 1164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:25.702737 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.702148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39904)
0808 15:55:25.702432 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.702558 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 15:55:25.703933 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.703516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39916)
0808 15:55:25.703687 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.703774 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:55:25.739675 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1042
I20260808 15:55:25.761251 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1172
I20260808 15:55:25.764535 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.764133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39930)
0808 15:55:25.764308 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.764397 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 15:55:25.765709 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.765332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39936)
0808 15:55:25.765527 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.765603 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 15:55:25.767676 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.762149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.27.254.3:45533)
0808 15:55:25.762318 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:25.762323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:25.762407 (+ 84us) server_negotiation.cc:407] Connection header received
0808 15:55:25.762613 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:25.762617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:25.762666 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:25.762750 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:25.766970 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:25.766974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:25.766977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:25.767001 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:25.767032 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:25.767042 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:25.767173 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:25.767327 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 15:55:25.767377 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:55:25.768435 1294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:25.846264 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.845731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39938)
0808 15:55:25.845963 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.846074 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 15:55:25.847397 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:25.846946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:34601 (local address 127.0.0.1:39950)
0808 15:55:25.847156 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:25.847246 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 15:55:25.887477 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1172
I20260808 15:55:25.908476 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 821
I20260808 15:55:25.909688 882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:26.017273 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 821
2026-08-08T15:55:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2701 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 15:55:26.057734 28664 mini_oidc.cc:152] Starting JWKS server
I20260808 15:55:26.061391 28664 webserver.cc:460] Webserver started at https://127.0.0.1:39199/ using document root <none> and password file <none>
I20260808 15:55:26.061821 28664 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:55:26.062501 28664 webserver.cc:460] Webserver started at http://127.0.0.1:42191/ using document root <none> and password file <none>
2026-08-08T15:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:26Z Disabled control of system clock
I20260808 15:55:26.088099 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:42117
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:34607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:42117
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39199/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/testchainca.cert with env {}
W20260808 15:55:26.399120 1354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:26.399467 1354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:26.399634 1354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:26.406605 1354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:26.406762 1354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:26.406852 1354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:26.406955 1354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:26.407039 1354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:55:26.407123 1354 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:55:26.407187 1354 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39199/jwks.json
I20260808 15:55:26.416417 1354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:42117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:42117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39199/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:26.417649 1354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:26.418905 1354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:26.430158 1362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:26.430321 1360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:26.430328 1359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:26.430917 1354 server_base.cc:1034] running on GCE node
I20260808 15:55:26.431689 1354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:26.433073 1354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:26.434290 1354 hybrid_clock.cc:648] HybridClock initialized: now 1786204526434251 us; error 58 us; skew 500 ppm
I20260808 15:55:26.434618 1354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:26.436685 1354 webserver.cc:460] Webserver started at http://127.27.254.62:46343/ using document root <none> and password file <none>
I20260808 15:55:26.437316 1354 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:26.437464 1354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:26.437770 1354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:26.440465 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "301baf1da97a45f7ad2ff8adde592d13"
format_stamp: "Formatted at 2026-08-08 15:55:26 on dist-test-slave-srgv"
server_key: "854a248f51398b6f2e1dd8e41bcfe2c7"
server_key_iv: "2a1de322544c99a952314ada89aab9ec"
server_key_version: "encryptionkey@0"
I20260808 15:55:26.441221 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "301baf1da97a45f7ad2ff8adde592d13"
format_stamp: "Formatted at 2026-08-08 15:55:26 on dist-test-slave-srgv"
server_key: "854a248f51398b6f2e1dd8e41bcfe2c7"
server_key_iv: "2a1de322544c99a952314ada89aab9ec"
server_key_version: "encryptionkey@0"
I20260808 15:55:26.445871 1354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 15:55:26.449066 1368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:26.450332 1354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:26.450519 1354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "301baf1da97a45f7ad2ff8adde592d13"
format_stamp: "Formatted at 2026-08-08 15:55:26 on dist-test-slave-srgv"
server_key: "854a248f51398b6f2e1dd8e41bcfe2c7"
server_key_iv: "2a1de322544c99a952314ada89aab9ec"
server_key_version: "encryptionkey@0"
I20260808 15:55:26.450740 1354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:26.465086 1354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:26.465898 1354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:26.466216 1354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:26.484313 1354 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:42117
I20260808 15:55:26.484330 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:42117 every 8 connection(s)
I20260808 15:55:26.485761 1354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:26.489526 1420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:26.495318 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1354
I20260808 15:55:26.495587 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:26.496359 28664 external_mini_cluster.cc:1428] Setting key af600ea57b13a1450437f2ce31e5c8ed
I20260808 15:55:26.500610 1420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13: Bootstrap starting.
I20260808 15:55:26.503702 1420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:26.504683 1420 log.cc:824] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:26.507705 1420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13: No bootstrap required, opened a new log
I20260808 15:55:26.510089 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:26.498399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.27.254.62:42117)
0808 15:55:26.499375 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:26.499385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:26.499425 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:55:26.501083 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:26.501104 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:26.501594 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:26.501939 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:26.502528 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:26.503740 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:26.505241 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:26.506056 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:26.506225 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:26.507326 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:26.507342 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:26.507358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:26.508028 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:26.508150 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:26.508191 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:26.508322 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:26.508494 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 15:55:26.508805 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260808 15:55:26.513819 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:55:26.516386 1420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301baf1da97a45f7ad2ff8adde592d13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42117 } }
I20260808 15:55:26.516784 1420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:26.516884 1420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301baf1da97a45f7ad2ff8adde592d13, State: Initialized, Role: FOLLOWER
I20260808 15:55:26.517510 1420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301baf1da97a45f7ad2ff8adde592d13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42117 } }
I20260808 15:55:26.517728 1420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:26.517853 1420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:26.518018 1420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:26.520133 1420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301baf1da97a45f7ad2ff8adde592d13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42117 } }
I20260808 15:55:26.520630 1420 leader_election.cc:302] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 301baf1da97a45f7ad2ff8adde592d13; no voters:
I20260808 15:55:26.521217 1420 leader_election.cc:288] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:26.521370 1425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:26.522107 1425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [term 1 LEADER]: Becoming Leader. State: Replica: 301baf1da97a45f7ad2ff8adde592d13, State: Running, Role: LEADER
I20260808 15:55:26.522652 1425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301baf1da97a45f7ad2ff8adde592d13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42117 } }
I20260808 15:55:26.523422 1420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:26.525525 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 301baf1da97a45f7ad2ff8adde592d13. Latest consensus state: current_term: 1 leader_uuid: "301baf1da97a45f7ad2ff8adde592d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301baf1da97a45f7ad2ff8adde592d13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42117 } } }
I20260808 15:55:26.525550 1426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "301baf1da97a45f7ad2ff8adde592d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301baf1da97a45f7ad2ff8adde592d13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 42117 } } }
I20260808 15:55:26.525959 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:26.526001 1426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13 [sys.catalog]: This master's current role is: LEADER
W20260808 15:55:26.534842 1441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:55:26.534971 1441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:55:26.535122 1432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:26.536692 1432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:26.544811 1432 catalog_manager.cc:1347] Generated new cluster ID: 1ddf29c026254c149a0b2961508fb8a6
I20260808 15:55:26.544919 1432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:26.558806 1432 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:26.560298 1432 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:26.567602 1432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 301baf1da97a45f7ad2ff8adde592d13: Generated new TSK 0
I20260808 15:55:26.568436 1432 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 15:55:26.602423 1450 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 15:55:26.602626 1450 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:55:26.604730 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:26.582252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.27.254.62:42117)
0808 15:55:26.582435 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:26.582441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:26.582863 (+ 422us) server_negotiation.cc:407] Connection header received
0808 15:55:26.583178 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:26.583184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:26.583305 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:26.583457 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:55:26.583962 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:26.585071 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:26.586466 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:26.587163 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:26.587289 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:26.587739 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:55:26.604124 (+ 16385us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39199/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 15:55:26.604463 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39199/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 15:55:26.604839 1423 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39199/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 15:55:26.606247 1449 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:26.582117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:42117 (local address 127.0.0.1:59366)
0808 15:55:26.582766 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:26.582782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:26.582915 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:26.583489 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:26.583492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:26.583502 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:55:26.583827 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:26.583835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:26.585203 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:26.585206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:26.586329 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:26.586341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:26.587308 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:26.587312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:26.587438 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:26.587491 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:55:26.604373 (+ 16882us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39199/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 15:55:26.604609 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.254.62:42117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39199/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":357,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.254.62:42117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39199/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 15:55:26.608232 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1354
I20260808 15:55:26.609298 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:26.699930 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1354
2026-08-08T15:55:26Z chronyd exiting
I20260808 15:55:26.749846 28664 test_util.cc:175] -----------------------------------------------
I20260808 15:55:26.750006 28664 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204485174398-28664-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 15:55:26.753671 28664 mini_oidc.cc:152] Starting JWKS server
I20260808 15:55:26.755461 28664 webserver.cc:460] Webserver started at https://127.0.0.1:39819/ using document root <none> and password file <none>
I20260808 15:55:26.755697 28664 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:55:26.756335 28664 webserver.cc:460] Webserver started at http://127.0.0.1:43461/ using document root <none> and password file <none>
2026-08-08T15:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:26Z Disabled control of system clock
I20260808 15:55:26.782509 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35621
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:35621
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39819/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/testchainca.cert with env {}
W20260808 15:55:27.092818 1457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:27.093096 1457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:27.093204 1457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:27.100366 1457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:27.100468 1457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:27.100507 1457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:27.100564 1457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:27.100608 1457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:55:27.100667 1457 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:55:27.100715 1457 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39819/jwks.json
I20260808 15:55:27.110581 1457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:35621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39819/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:27.111770 1457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:27.112977 1457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:27.124413 1465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:27.124377 1463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:27.124446 1462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:27.125140 1457 server_base.cc:1034] running on GCE node
I20260808 15:55:27.125931 1457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:27.127323 1457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:27.128525 1457 hybrid_clock.cc:648] HybridClock initialized: now 1786204527128496 us; error 33 us; skew 500 ppm
I20260808 15:55:27.128880 1457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:27.131151 1457 webserver.cc:460] Webserver started at http://127.27.254.62:39963/ using document root <none> and password file <none>
I20260808 15:55:27.131810 1457 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:27.131927 1457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:27.132269 1457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:27.135162 1457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "eee1c3a707744e0eae592ddb808a2e61"
format_stamp: "Formatted at 2026-08-08 15:55:27 on dist-test-slave-srgv"
server_key: "5d9f0424784608b93ceb3562379e8585"
server_key_iv: "45729d81232b01f92de10a86bff5b1b3"
server_key_version: "encryptionkey@0"
I20260808 15:55:27.135918 1457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "eee1c3a707744e0eae592ddb808a2e61"
format_stamp: "Formatted at 2026-08-08 15:55:27 on dist-test-slave-srgv"
server_key: "5d9f0424784608b93ceb3562379e8585"
server_key_iv: "45729d81232b01f92de10a86bff5b1b3"
server_key_version: "encryptionkey@0"
I20260808 15:55:27.140902 1457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:55:27.144101 1471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:27.145454 1457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:55:27.145640 1457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "eee1c3a707744e0eae592ddb808a2e61"
format_stamp: "Formatted at 2026-08-08 15:55:27 on dist-test-slave-srgv"
server_key: "5d9f0424784608b93ceb3562379e8585"
server_key_iv: "45729d81232b01f92de10a86bff5b1b3"
server_key_version: "encryptionkey@0"
I20260808 15:55:27.145845 1457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:27.193336 1457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:27.194206 1457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:27.194464 1457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:27.212177 1457 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:35621
I20260808 15:55:27.212213 1522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:35621 every 8 connection(s)
I20260808 15:55:27.213511 1457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:27.217206 1523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:27.218892 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1457
I20260808 15:55:27.219112 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:27.219916 28664 external_mini_cluster.cc:1428] Setting key 77b52e0e526c229316c11f481db4afaf
I20260808 15:55:27.230082 1523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61: Bootstrap starting.
I20260808 15:55:27.232721 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:27.221916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.27.254.62:35621)
0808 15:55:27.222922 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:27.222932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:27.222970 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:55:27.224504 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:27.224524 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:27.224964 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:27.225332 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:27.225890 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.227102 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.228432 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.229074 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.229271 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:27.230547 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:27.230554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:27.230566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:27.231075 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:27.231170 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:27.231190 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:27.231315 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:27.231553 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 15:55:27.231843 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":388,"threads_started":1}
I20260808 15:55:27.233384 1523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:27.234688 1523 log.cc:824] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:27.236810 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:55:27.237854 1523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61: No bootstrap required, opened a new log
I20260808 15:55:27.245585 1523 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee1c3a707744e0eae592ddb808a2e61" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35621 } }
I20260808 15:55:27.245927 1523 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:27.246047 1523 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee1c3a707744e0eae592ddb808a2e61, State: Initialized, Role: FOLLOWER
I20260808 15:55:27.246569 1523 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee1c3a707744e0eae592ddb808a2e61" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35621 } }
I20260808 15:55:27.246809 1523 raft_consensus.cc:388] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:27.246959 1523 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:27.247118 1523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:27.249578 1523 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee1c3a707744e0eae592ddb808a2e61" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35621 } }
I20260808 15:55:27.250141 1523 leader_election.cc:302] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eee1c3a707744e0eae592ddb808a2e61; no voters:
I20260808 15:55:27.250648 1523 leader_election.cc:288] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:27.250804 1528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:27.251482 1528 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [term 1 LEADER]: Becoming Leader. State: Replica: eee1c3a707744e0eae592ddb808a2e61, State: Running, Role: LEADER
I20260808 15:55:27.252019 1528 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee1c3a707744e0eae592ddb808a2e61" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35621 } }
I20260808 15:55:27.252635 1523 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:27.254783 1529 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eee1c3a707744e0eae592ddb808a2e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee1c3a707744e0eae592ddb808a2e61" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35621 } } }
I20260808 15:55:27.255185 1529 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:27.254805 1530 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee1c3a707744e0eae592ddb808a2e61. Latest consensus state: current_term: 1 leader_uuid: "eee1c3a707744e0eae592ddb808a2e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee1c3a707744e0eae592ddb808a2e61" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35621 } } }
I20260808 15:55:27.255597 1530 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:27.262600 1534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:27.267661 1534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:27.276278 1534 catalog_manager.cc:1347] Generated new cluster ID: caaec6a75f2e476eb39c0746e0196d18
I20260808 15:55:27.276391 1534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:27.287806 1534 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:27.289141 1534 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:27.295689 1534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eee1c3a707744e0eae592ddb808a2e61: Generated new TSK 0
I20260808 15:55:27.296479 1534 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 15:55:27.332367 1553 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 15:55:27.332486 1553 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:55:27.333720 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:27.314200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.27.254.62:35621)
0808 15:55:27.314387 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:27.314393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:27.314794 (+ 401us) server_negotiation.cc:407] Connection header received
0808 15:55:27.315015 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:27.315021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:27.315105 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:27.315265 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:55:27.315716 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.316806 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.318315 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.318788 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.318935 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:27.319452 (+ 517us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:55:27.332980 (+ 13528us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 15:55:27.333358 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43780: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 15:55:27.333858 1526 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43780: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 15:55:27.333880 1552 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:27.314058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35621 (local address 127.0.0.1:43780)
0808 15:55:27.314695 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:27.314714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:27.314823 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:27.315278 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:27.315281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:27.315289 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:55:27.315538 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:27.315546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.316975 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.316979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:27.318174 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:27.318185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.319026 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.319031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:27.319175 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:27.319207 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:55:27.333196 (+ 13989us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 15:55:27.333476 (+ 280us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.254.62:35621: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39819/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":379,"threads_started":1}
I20260808 15:55:27.335114 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1457
I20260808 15:55:27.336239 1518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:27.426352 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1457
2026-08-08T15:55:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (727 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 15:55:27.481792 28664 mini_oidc.cc:152] Starting JWKS server
I20260808 15:55:27.483606 28664 webserver.cc:460] Webserver started at https://127.0.0.1:38181/ using document root <none> and password file <none>
I20260808 15:55:27.483861 28664 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:55:27.484472 28664 webserver.cc:460] Webserver started at http://127.0.0.1:45247/ using document root <none> and password file <none>
2026-08-08T15:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:27Z Disabled control of system clock
I20260808 15:55:27.510661 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:34973
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:34973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38181/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 15:55:27.820294 1560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:27.820570 1560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:27.820753 1560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:27.827877 1560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:27.827973 1560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:27.828011 1560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:27.828089 1560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:27.828178 1560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:55:27.828290 1560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:55:27.828344 1560 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38181/jwks.json
I20260808 15:55:27.837430 1560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:34973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38181/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:27.838724 1560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:27.840505 1560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:27.851883 1568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:27.851831 1566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:27.851902 1565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:27.853222 1560 server_base.cc:1034] running on GCE node
I20260808 15:55:27.854110 1560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:27.855394 1560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:27.856629 1560 hybrid_clock.cc:648] HybridClock initialized: now 1786204527856591 us; error 56 us; skew 500 ppm
I20260808 15:55:27.856995 1560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:27.859220 1560 webserver.cc:460] Webserver started at http://127.27.254.62:44241/ using document root <none> and password file <none>
I20260808 15:55:27.859899 1560 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:27.860031 1560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:27.860357 1560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:27.863256 1560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "f009721ebbdd47e8851600e263f998ee"
format_stamp: "Formatted at 2026-08-08 15:55:27 on dist-test-slave-srgv"
server_key: "ef190b6006893dbd5f28b264d9fbe80b"
server_key_iv: "60afb40d3f326aa94947d47439790d7b"
server_key_version: "encryptionkey@0"
I20260808 15:55:27.864104 1560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "f009721ebbdd47e8851600e263f998ee"
format_stamp: "Formatted at 2026-08-08 15:55:27 on dist-test-slave-srgv"
server_key: "ef190b6006893dbd5f28b264d9fbe80b"
server_key_iv: "60afb40d3f326aa94947d47439790d7b"
server_key_version: "encryptionkey@0"
I20260808 15:55:27.869102 1560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:55:27.872354 1574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:27.873649 1560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:27.873826 1560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "f009721ebbdd47e8851600e263f998ee"
format_stamp: "Formatted at 2026-08-08 15:55:27 on dist-test-slave-srgv"
server_key: "ef190b6006893dbd5f28b264d9fbe80b"
server_key_iv: "60afb40d3f326aa94947d47439790d7b"
server_key_version: "encryptionkey@0"
I20260808 15:55:27.874042 1560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:27.890520 1560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:27.891314 1560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:27.891633 1560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:27.909523 1560 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:34973
I20260808 15:55:27.909500 1625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:34973 every 8 connection(s)
I20260808 15:55:27.910857 1560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:27.914472 1626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:27.917438 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1560
I20260808 15:55:27.917661 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:27.918428 28664 external_mini_cluster.cc:1428] Setting key c533214a2ca317977502984ef3d1c221
I20260808 15:55:27.926597 1626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee: Bootstrap starting.
I20260808 15:55:27.930135 1626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:27.931185 1626 log.cc:824] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:27.931643 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:27.920356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.27.254.62:34973)
0808 15:55:27.921271 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:27.921281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:27.921320 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:55:27.922949 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:27.922969 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:27.923411 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:27.923743 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:27.924286 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.925573 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.927280 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:27.927896 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:27.928027 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:27.929239 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:27.929246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:27.929258 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:27.929831 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:27.929914 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:27.929938 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:27.930027 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:27.930345 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 15:55:27.930680 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":371,"threads_started":1}
I20260808 15:55:27.934311 1626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee: No bootstrap required, opened a new log
I20260808 15:55:27.935631 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:55:27.942744 1626 raft_consensus.cc:348] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f009721ebbdd47e8851600e263f998ee" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 34973 } }
I20260808 15:55:27.943094 1626 raft_consensus.cc:374] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:27.943239 1626 raft_consensus.cc:729] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f009721ebbdd47e8851600e263f998ee, State: Initialized, Role: FOLLOWER
I20260808 15:55:27.943926 1626 consensus_queue.cc:260] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f009721ebbdd47e8851600e263f998ee" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 34973 } }
I20260808 15:55:27.944219 1626 raft_consensus.cc:388] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:27.944362 1626 raft_consensus.cc:482] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:27.944535 1626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:27.946776 1626 raft_consensus.cc:504] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f009721ebbdd47e8851600e263f998ee" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 34973 } }
I20260808 15:55:27.947275 1626 leader_election.cc:302] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f009721ebbdd47e8851600e263f998ee; no voters:
I20260808 15:55:27.948482 1626 leader_election.cc:288] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:27.948611 1631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:27.950269 1631 raft_consensus.cc:686] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [term 1 LEADER]: Becoming Leader. State: Replica: f009721ebbdd47e8851600e263f998ee, State: Running, Role: LEADER
I20260808 15:55:27.950950 1626 sys_catalog.cc:564] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:27.950824 1631 consensus_queue.cc:237] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f009721ebbdd47e8851600e263f998ee" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 34973 } }
I20260808 15:55:27.956105 1633 sys_catalog.cc:455] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f009721ebbdd47e8851600e263f998ee. Latest consensus state: current_term: 1 leader_uuid: "f009721ebbdd47e8851600e263f998ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f009721ebbdd47e8851600e263f998ee" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 34973 } } }
I20260808 15:55:27.956058 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f009721ebbdd47e8851600e263f998ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f009721ebbdd47e8851600e263f998ee" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 34973 } } }
I20260808 15:55:27.956498 1633 sys_catalog.cc:458] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:27.956511 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:27.958166 1641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:27.963646 1641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:27.972901 1641 catalog_manager.cc:1347] Generated new cluster ID: fef6732af9ee4a8c9ab0d8d232e1c0dd
I20260808 15:55:27.973029 1641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:27.985625 1641 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:27.986881 1641 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:27.995338 1641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f009721ebbdd47e8851600e263f998ee: Generated new TSK 0
I20260808 15:55:27.996217 1641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:55:28.024268 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:28.016138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.27.254.62:34973)
0808 15:55:28.016324 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:28.016330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:28.016702 (+ 372us) server_negotiation.cc:407] Connection header received
0808 15:55:28.016908 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:28.016913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:28.016997 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:28.017188 (+ 191us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:55:28.017674 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.018841 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.020170 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.020832 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.020955 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:28.021479 (+ 524us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:55:28.023663 (+ 2184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38181/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 15:55:28.023955 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38181/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 15:55:28.024402 1629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38181/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 15:55:28.025928 1655 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:28.015980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:34973 (local address 127.0.0.1:54078)
0808 15:55:28.016599 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:28.016614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:28.016731 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:28.017201 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:28.017205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:28.017214 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:55:28.017524 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:28.017534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.018973 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.018977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:28.020021 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:28.020031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.021000 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.021004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:28.021137 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:28.021175 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:55:28.023856 (+ 2681us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38181/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 15:55:28.025506 (+ 1650us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.254.62:34973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38181/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1359,"thread_start_us":357,"threads_started":1}
I20260808 15:55:28.026852 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1560
I20260808 15:55:28.028040 1621 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 15:55:28.036275 1656 webserver.cc:533] Webserver: sslize failed:
E20260808 15:55:28.036398 1656 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:55:28.116259 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1560
2026-08-08T15:55:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (688 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T15:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:28Z Disabled control of system clock
I20260808 15:55:28.195878 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:45833
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:45833
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/testchainca.cert with env {}
W20260808 15:55:28.503738 1661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:28.504012 1661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:28.504105 1661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:28.510840 1661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:28.511003 1661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:28.511053 1661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:28.511116 1661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:28.511158 1661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:55:28.511216 1661 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:55:28.511253 1661 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 15:55:28.520484 1661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:45833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:45833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:28.521764 1661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:28.522949 1661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:28.534288 1666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:28.534366 1667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:28.534447 1669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:28.535362 1661 server_base.cc:1034] running on GCE node
I20260808 15:55:28.536041 1661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:28.537360 1661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:28.538601 1661 hybrid_clock.cc:648] HybridClock initialized: now 1786204528538545 us; error 54 us; skew 500 ppm
I20260808 15:55:28.538928 1661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:55:28.541138 1661 webserver.cc:460] Webserver started at http://127.27.254.62:35677/ using document root <none> and password file <none>
I20260808 15:55:28.541898 1661 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:28.542019 1661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:28.542316 1661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:28.544909 1661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "1e20d3ff352d48f597add0318f8c4828"
format_stamp: "Formatted at 2026-08-08 15:55:28 on dist-test-slave-srgv"
server_key: "8c561c99f21c0358f87039746fbe8645"
server_key_iv: "8f2f7f666d963063a650318dc1d8501d"
server_key_version: "encryptionkey@0"
I20260808 15:55:28.545724 1661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "1e20d3ff352d48f597add0318f8c4828"
format_stamp: "Formatted at 2026-08-08 15:55:28 on dist-test-slave-srgv"
server_key: "8c561c99f21c0358f87039746fbe8645"
server_key_iv: "8f2f7f666d963063a650318dc1d8501d"
server_key_version: "encryptionkey@0"
I20260808 15:55:28.550423 1661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:55:28.553546 1675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:28.554790 1661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 15:55:28.554955 1661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "1e20d3ff352d48f597add0318f8c4828"
format_stamp: "Formatted at 2026-08-08 15:55:28 on dist-test-slave-srgv"
server_key: "8c561c99f21c0358f87039746fbe8645"
server_key_iv: "8f2f7f666d963063a650318dc1d8501d"
server_key_version: "encryptionkey@0"
I20260808 15:55:28.555153 1661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:28.572288 1661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:28.573117 1661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:28.573407 1661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:28.591341 1661 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:45833
I20260808 15:55:28.591360 1726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:45833 every 8 connection(s)
I20260808 15:55:28.592682 1661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:28.596306 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:28.602767 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1661
I20260808 15:55:28.602977 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:28.603600 28664 external_mini_cluster.cc:1428] Setting key a67c36b3d8362972d25a135e4594ac6f
I20260808 15:55:28.606722 1727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828: Bootstrap starting.
I20260808 15:55:28.610978 1727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:28.612294 1727 log.cc:824] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:28.614949 1727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828: No bootstrap required, opened a new log
I20260808 15:55:28.618628 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:28.605472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.27.254.62:45833)
0808 15:55:28.606540 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:28.606554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:28.606604 (+ 50us) server_negotiation.cc:407] Connection header received
0808 15:55:28.609211 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:28.609245 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:28.609855 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:28.610312 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:28.610850 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.612888 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.614280 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.614942 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.615147 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:28.616367 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:28.616375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:28.616388 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:55:28.616934 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:28.617015 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:55:28.617038 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:28.617179 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:28.617432 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 15:55:28.617721 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":317,"threads_started":1}
I20260808 15:55:28.622448 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:55:28.623468 1727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e20d3ff352d48f597add0318f8c4828" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45833 } }
I20260808 15:55:28.623817 1727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:28.623922 1727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e20d3ff352d48f597add0318f8c4828, State: Initialized, Role: FOLLOWER
I20260808 15:55:28.624431 1727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e20d3ff352d48f597add0318f8c4828" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45833 } }
I20260808 15:55:28.624706 1727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:28.624863 1727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:28.625048 1727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:28.627447 1727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e20d3ff352d48f597add0318f8c4828" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45833 } }
I20260808 15:55:28.627861 1727 leader_election.cc:302] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e20d3ff352d48f597add0318f8c4828; no voters:
I20260808 15:55:28.628466 1727 leader_election.cc:288] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:28.628600 1732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:28.629678 1732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [term 1 LEADER]: Becoming Leader. State: Replica: 1e20d3ff352d48f597add0318f8c4828, State: Running, Role: LEADER
I20260808 15:55:28.630184 1732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e20d3ff352d48f597add0318f8c4828" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45833 } }
I20260808 15:55:28.630569 1727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:28.632876 1733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e20d3ff352d48f597add0318f8c4828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e20d3ff352d48f597add0318f8c4828" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45833 } } }
I20260808 15:55:28.633262 1733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:28.633781 1734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e20d3ff352d48f597add0318f8c4828. Latest consensus state: current_term: 1 leader_uuid: "1e20d3ff352d48f597add0318f8c4828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e20d3ff352d48f597add0318f8c4828" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45833 } } }
I20260808 15:55:28.634116 1734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:28.635465 1740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:28.640482 1740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:28.649200 1740 catalog_manager.cc:1347] Generated new cluster ID: 8964174e50df4e988dc9ba85338611a5
I20260808 15:55:28.649300 1740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:28.658977 1740 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:28.660277 1740 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:28.671146 1740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e20d3ff352d48f597add0318f8c4828: Generated new TSK 0
I20260808 15:55:28.671953 1740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:55:28.735168 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:28.683455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.27.254.62:45833)
0808 15:55:28.683604 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:28.683608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:28.684080 (+ 472us) server_negotiation.cc:407] Connection header received
0808 15:55:28.684359 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:28.684364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:28.684428 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:28.684589 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:55:28.685034 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.685819 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.687180 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.687610 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.687766 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:28.688211 (+ 445us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:55:28.734382 (+ 46171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 15:55:28.734807 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45788: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 15:55:28.735438 1730 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45788: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 15:55:28.735456 1756 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:28.683459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:45833 (local address 127.0.0.1:45788)
0808 15:55:28.683979 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:28.683994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:28.684152 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:28.684616 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:28.684621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:28.684635 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:55:28.684891 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:28.684899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.685960 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.685970 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:28.687041 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:28.687052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:28.687794 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:28.687800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:28.687950 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:28.687981 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:55:28.734666 (+ 46685us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 15:55:28.734981 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.254.62:45833: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":288,"threads_started":1}
I20260808 15:55:28.736738 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1661
I20260808 15:55:28.737957 1722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:28.834103 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1661
2026-08-08T15:55:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (698 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (404 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (410 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:29 dist-test-slave-srgv krb5kdc[1771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:29 dist-test-slave-srgv krb5kdc[1771](info): set up 2 sockets
Aug 08 15:55:29 dist-test-slave-srgv krb5kdc[1771](info): commencing operation
krb5kdc: starting...
W20260808 15:55:30.735926 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:30 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:30.944871 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:44409
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:31.262585 1784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:31.262883 1784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:31.263054 1784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:31.270207 1784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:31.270314 1784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:31.270434 1784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:31.270535 1784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:31.270622 1784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:31.280571 1784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:44409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:31.281950 1784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:31.283298 1784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:31.295207 1790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:31.295478 1789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:31.295251 1792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:31.296272 1784 server_base.cc:1034] running on GCE node
I20260808 15:55:31.297066 1784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:31.298463 1784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:31.299671 1784 hybrid_clock.cc:648] HybridClock initialized: now 1786204531299626 us; error 30 us; skew 500 ppm
I20260808 15:55:31.300036 1784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:31 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204531, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:31.306195 1784 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:31.307699 1784 webserver.cc:460] Webserver started at http://127.27.254.62:46469/ using document root <none> and password file <none>
I20260808 15:55:31.308370 1784 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:31.308514 1784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:31.308835 1784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:31.311786 1784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "c66b0f6926ac407f83880eeed8771575"
format_stamp: "Formatted at 2026-08-08 15:55:31 on dist-test-slave-srgv"
server_key: "e1d92d4f146866a1ddd9d43a4f25d72c"
server_key_iv: "d73f8539e159ba0aeec1bf982a1f3278"
server_key_version: "encryptionkey@0"
I20260808 15:55:31.312577 1784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "c66b0f6926ac407f83880eeed8771575"
format_stamp: "Formatted at 2026-08-08 15:55:31 on dist-test-slave-srgv"
server_key: "e1d92d4f146866a1ddd9d43a4f25d72c"
server_key_iv: "d73f8539e159ba0aeec1bf982a1f3278"
server_key_version: "encryptionkey@0"
I20260808 15:55:31.317605 1784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:55:31.320813 1799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:31.322099 1784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:31.322284 1784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "c66b0f6926ac407f83880eeed8771575"
format_stamp: "Formatted at 2026-08-08 15:55:31 on dist-test-slave-srgv"
server_key: "e1d92d4f146866a1ddd9d43a4f25d72c"
server_key_iv: "d73f8539e159ba0aeec1bf982a1f3278"
server_key_version: "encryptionkey@0"
I20260808 15:55:31.322480 1784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:31.347775 1784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:31.351747 1784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:31.352044 1784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:31.370384 1784 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:44409
I20260808 15:55:31.370420 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:44409 every 8 connection(s)
I20260808 15:55:31.371793 1784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:31.374913 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1784
I20260808 15:55:31.375123 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:31.376003 28664 external_mini_cluster.cc:1428] Setting key cbf307653e424c8bf7f3fe10650ffd06
I20260808 15:55:31.375998 1851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:31.390377 1851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575: Bootstrap starting.
Aug 08 15:55:31 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:31.394119 1851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:31.395427 1851 log.cc:824] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:31.398895 1851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575: No bootstrap required, opened a new log
I20260808 15:55:31.400322 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:31.378088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.27.254.62:44409)
0808 15:55:31.378868 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:31.378880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:31.378923 (+ 43us) server_negotiation.cc:407] Connection header received
0808 15:55:31.380742 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:31.380762 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:31.381162 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:31.381566 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:31.383181 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:31.384478 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:31.386296 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:31.386977 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:31.387125 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:31.391866 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:31.391877 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:31.391896 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:31.391941 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:31.395783 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:31.396689 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:31.396695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:31.396700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:31.396891 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:31.397268 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:31.397272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:31.397275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:31.397950 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:31.398260 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:31.398565 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 15:55:31.398990 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
I20260808 15:55:31.406994 1851 raft_consensus.cc:348] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66b0f6926ac407f83880eeed8771575" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 44409 } }
I20260808 15:55:31.407296 1851 raft_consensus.cc:374] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:31.407419 1851 raft_consensus.cc:729] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66b0f6926ac407f83880eeed8771575, State: Initialized, Role: FOLLOWER
I20260808 15:55:31.407889 1851 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66b0f6926ac407f83880eeed8771575" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 44409 } }
I20260808 15:55:31.408087 1851 raft_consensus.cc:388] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:31.408177 1851 raft_consensus.cc:482] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:31.408296 1851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:31.410770 1851 raft_consensus.cc:504] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66b0f6926ac407f83880eeed8771575" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 44409 } }
I20260808 15:55:31.411201 1851 leader_election.cc:302] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66b0f6926ac407f83880eeed8771575; no voters:
I20260808 15:55:31.411713 1851 leader_election.cc:288] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:31.411857 1856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:31.413029 1856 raft_consensus.cc:686] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [term 1 LEADER]: Becoming Leader. State: Replica: c66b0f6926ac407f83880eeed8771575, State: Running, Role: LEADER
I20260808 15:55:31.413647 1856 consensus_queue.cc:237] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66b0f6926ac407f83880eeed8771575" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 44409 } }
I20260808 15:55:31.414065 1851 sys_catalog.cc:564] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:31.416734 1857 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c66b0f6926ac407f83880eeed8771575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66b0f6926ac407f83880eeed8771575" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 44409 } } }
I20260808 15:55:31.417217 1857 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:31.416723 1858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c66b0f6926ac407f83880eeed8771575. Latest consensus state: current_term: 1 leader_uuid: "c66b0f6926ac407f83880eeed8771575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66b0f6926ac407f83880eeed8771575" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 44409 } } }
I20260808 15:55:31.417393 1858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:31.418521 1862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:31.422961 1862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:31.431994 1862 catalog_manager.cc:1347] Generated new cluster ID: 6ac9a44f0b494a1da2767af460fa05cc
I20260808 15:55:31.432156 1862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:31.441854 1862 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:31.443152 1862 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:31.452138 1862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c66b0f6926ac407f83880eeed8771575: Generated new TSK 0
I20260808 15:55:31.453007 1862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:55:31.498926 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:44409
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:31.841372 1877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:31.841720 1877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:31.841892 1877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:31.849470 1877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:31.849609 1877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:31.849805 1877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:31.860103 1877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:31.861508 1877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:31.862833 1877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:31.875900 1882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:31.875861 1883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:31.875949 1885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:31.876535 1877 server_base.cc:1034] running on GCE node
I20260808 15:55:31.877091 1877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:31.878049 1877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:31.879269 1877 hybrid_clock.cc:648] HybridClock initialized: now 1786204531879221 us; error 42 us; skew 500 ppm
I20260808 15:55:31.879571 1877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:31 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204531, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:31.882627 1877 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:55:31.883972 1877 webserver.cc:460] Webserver started at http://127.27.254.1:33651/ using document root <none> and password file <none>
I20260808 15:55:31.884666 1877 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:31.884810 1877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:31.885130 1877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:31.888060 1877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "451ea4f4497244e8b736fab7f8f9fb3e"
format_stamp: "Formatted at 2026-08-08 15:55:31 on dist-test-slave-srgv"
server_key: "7f9e5e0640fc145d0e0650f37ede8c75"
server_key_iv: "924b101c8ea99bd21c53b4700d19e252"
server_key_version: "encryptionkey@0"
I20260808 15:55:31.888839 1877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "451ea4f4497244e8b736fab7f8f9fb3e"
format_stamp: "Formatted at 2026-08-08 15:55:31 on dist-test-slave-srgv"
server_key: "7f9e5e0640fc145d0e0650f37ede8c75"
server_key_iv: "924b101c8ea99bd21c53b4700d19e252"
server_key_version: "encryptionkey@0"
I20260808 15:55:31.894187 1877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 15:55:31.897691 1892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:31.899005 1877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:55:31.899171 1877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "451ea4f4497244e8b736fab7f8f9fb3e"
format_stamp: "Formatted at 2026-08-08 15:55:31 on dist-test-slave-srgv"
server_key: "7f9e5e0640fc145d0e0650f37ede8c75"
server_key_iv: "924b101c8ea99bd21c53b4700d19e252"
server_key_version: "encryptionkey@0"
I20260808 15:55:31.899385 1877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:31.910224 1877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:31.913982 1877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:31.914377 1877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:31.915356 1877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:31.916644 1877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:31.916719 1877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:31.916859 1877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:31.916940 1877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:31.950794 1877 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:38357
I20260808 15:55:31.950898 2004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:38357 every 8 connection(s)
I20260808 15:55:31.952494 1877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:31.958930 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 1877
I20260808 15:55:31.959160 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:31.959780 28664 external_mini_cluster.cc:1428] Setting key 55b4742c6ad63e77242c7ad954f4a65f
Aug 08 15:55:31 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204531, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:31.973517 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:31.955176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:60547 (local address 127.27.254.62:44409)
0808 15:55:31.955602 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:31.955607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:31.956646 (+ 1039us) server_negotiation.cc:407] Connection header received
0808 15:55:31.958548 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:31.958556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:31.958652 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:31.958795 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:31.960405 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:31.961281 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:31.962990 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:31.963749 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:31.963905 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:31.967291 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:31.967300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:31.967308 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:31.967355 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:31.970046 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:31.971075 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:31.971079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:31.971082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:31.971205 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:31.971703 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:31.971706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:31.971708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:31.971962 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:31.972082 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:31.973000 (+ 918us) server_negotiation.cc:299] Negotiation successful
0808 15:55:31.973183 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260808 15:55:31.975034 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:31.955804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:44409 (local address 127.27.254.1:60547)
0808 15:55:31.956493 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:31.956529 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:31.958244 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:31.958976 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:31.958985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:31.959595 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:31.960223 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:31.960241 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:31.961472 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:31.961476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:31.962833 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:31.962850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:31.963958 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:31.963963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:31.964165 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:31.964842 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:31.964866 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:31.967066 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:31.970180 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:31.970191 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:31.970211 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:31.970867 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:31.971320 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:31.971328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:31.971334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:31.971577 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:31.972092 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:31.972101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:31.972560 (+ 459us) client_negotiation.cc:769] Sending connection context
0808 15:55:31.972874 (+ 314us) client_negotiation.cc:240] Negotiation successful
0808 15:55:31.973500 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":276,"threads_started":1}
I20260808 15:55:31.976647 2005 heartbeater.cc:347] Connected to a master server at 127.27.254.62:44409
I20260808 15:55:31.976996 2005 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:31.977769 2005 heartbeater.cc:511] Master 127.27.254.62:44409 requested a full tablet report, sending...
I20260808 15:55:31.979895 1816 ts_manager.cc:194] Registered new tserver with Master: 451ea4f4497244e8b736fab7f8f9fb3e (127.27.254.1:38357)
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:55:31.981398 1816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:60547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:32.008803 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:44409
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:32.334453 2011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:32.334729 2011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:32.334878 2011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:32.342196 2011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:32.342305 2011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:32.342528 2011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:32.352598 2011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:32.354013 2011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:32.355301 2011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:32.375684 2017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:32.375684 2016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:32.376514 2019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:32.377789 2011 server_base.cc:1034] running on GCE node
I20260808 15:55:32.378481 2011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:32.379571 2011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:32.380854 2011 hybrid_clock.cc:648] HybridClock initialized: now 1786204532380778 us; error 78 us; skew 500 ppm
I20260808 15:55:32.381201 2011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:32 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204532, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:32.384425 2011 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:32.386621 2011 webserver.cc:460] Webserver started at http://127.27.254.2:33575/ using document root <none> and password file <none>
I20260808 15:55:32.387351 2011 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:32.387537 2011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:32.387882 2011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:32.390863 2011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "4d87173e706c40ba8454919330bf8155"
format_stamp: "Formatted at 2026-08-08 15:55:32 on dist-test-slave-srgv"
server_key: "fc186f7bf0ab1cbd140fec56176f4812"
server_key_iv: "f2b958bcd97568892aabff972157135d"
server_key_version: "encryptionkey@0"
I20260808 15:55:32.391799 2011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "4d87173e706c40ba8454919330bf8155"
format_stamp: "Formatted at 2026-08-08 15:55:32 on dist-test-slave-srgv"
server_key: "fc186f7bf0ab1cbd140fec56176f4812"
server_key_iv: "f2b958bcd97568892aabff972157135d"
server_key_version: "encryptionkey@0"
I20260808 15:55:32.397035 2011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:55:32.400635 2026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:32.402091 2011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:32.402253 2011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "4d87173e706c40ba8454919330bf8155"
format_stamp: "Formatted at 2026-08-08 15:55:32 on dist-test-slave-srgv"
server_key: "fc186f7bf0ab1cbd140fec56176f4812"
server_key_iv: "f2b958bcd97568892aabff972157135d"
server_key_version: "encryptionkey@0"
I20260808 15:55:32.402467 2011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:32.422583 2011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:32.427222 2011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:32.427539 2011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:32.428668 2011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:32.430209 2011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:32.430311 2011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:32.430467 2011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:32.430539 2011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:32.470345 2011 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:42573
I20260808 15:55:32.470398 2138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:42573 every 8 connection(s)
I20260808 15:55:32.471786 2011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:32.479542 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2011
I20260808 15:55:32.479754 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:32.480237 28664 external_mini_cluster.cc:1428] Setting key d6324551da8136973e25c67c3d456238
Aug 08 15:55:32 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204532, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:32.493302 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:32.474512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44433 (local address 127.27.254.62:44409)
0808 15:55:32.474980 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:32.474985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:32.475926 (+ 941us) server_negotiation.cc:407] Connection header received
0808 15:55:32.477837 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:32.477843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:32.477933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:32.478094 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:32.479879 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:32.480666 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:32.482757 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:32.483178 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:32.483276 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:32.487483 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:32.487488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:32.487491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:32.487530 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:32.490228 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:32.490970 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:32.490980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:32.490982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:32.491107 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:32.491562 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:32.491566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:32.491568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:32.491849 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:32.492005 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:32.492704 (+ 699us) server_negotiation.cc:299] Negotiation successful
0808 15:55:32.492924 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260808 15:55:32.494459 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:32.475039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:44409 (local address 127.27.254.2:44433)
0808 15:55:32.475768 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:32.475805 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:32.477566 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:32.478327 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:32.478335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:32.478852 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:32.479659 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:32.479687 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:32.480845 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:32.480849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:32.482595 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:32.482609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:32.483411 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:32.483422 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:32.483774 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:32.484517 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:32.484559 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:32.487246 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:32.490375 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:32.490384 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:32.490402 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:32.490839 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:32.491219 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:32.491224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:32.491228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:32.491422 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:32.491987 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:32.491996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:32.492334 (+ 338us) client_negotiation.cc:769] Sending connection context
0808 15:55:32.492599 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 15:55:32.493170 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260808 15:55:32.495961 2139 heartbeater.cc:347] Connected to a master server at 127.27.254.62:44409
I20260808 15:55:32.496279 2139 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:32.496997 2139 heartbeater.cc:511] Master 127.27.254.62:44409 requested a full tablet report, sending...
I20260808 15:55:32.498536 1816 ts_manager.cc:194] Registered new tserver with Master: 4d87173e706c40ba8454919330bf8155 (127.27.254.2:42573)
I20260808 15:55:32.499769 1816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:44433
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:32.528004 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:44409
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:32.856755 2145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:32.857025 2145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:32.857126 2145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:32.864326 2145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:32.864440 2145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:32.864553 2145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:32.874593 2145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:32.875846 2145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:32.877213 2145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:32.890159 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:32.890208 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:32.890450 2153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:32.891444 2145 server_base.cc:1034] running on GCE node
I20260808 15:55:32.892064 2145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:32.893055 2145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:32.894250 2145 hybrid_clock.cc:648] HybridClock initialized: now 1786204532894185 us; error 55 us; skew 500 ppm
I20260808 15:55:32.894526 2145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:32 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204532, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:32.897589 2145 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:32.899055 2145 webserver.cc:460] Webserver started at http://127.27.254.3:39113/ using document root <none> and password file <none>
I20260808 15:55:32.899736 2145 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:32.899899 2145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:32.900224 2145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:32.903164 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "93ec382379d64fbe99c4218a7a860abf"
format_stamp: "Formatted at 2026-08-08 15:55:32 on dist-test-slave-srgv"
server_key: "a326e8bdbb3cbcf65dd1c5f0fdf8196f"
server_key_iv: "495db57a29c790f045ee04be95418507"
server_key_version: "encryptionkey@0"
I20260808 15:55:32.903889 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "93ec382379d64fbe99c4218a7a860abf"
format_stamp: "Formatted at 2026-08-08 15:55:32 on dist-test-slave-srgv"
server_key: "a326e8bdbb3cbcf65dd1c5f0fdf8196f"
server_key_iv: "495db57a29c790f045ee04be95418507"
server_key_version: "encryptionkey@0"
I20260808 15:55:32.908967 2145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:55:32.912257 2160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:32.913616 2145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:32.913739 2145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "93ec382379d64fbe99c4218a7a860abf"
format_stamp: "Formatted at 2026-08-08 15:55:32 on dist-test-slave-srgv"
server_key: "a326e8bdbb3cbcf65dd1c5f0fdf8196f"
server_key_iv: "495db57a29c790f045ee04be95418507"
server_key_version: "encryptionkey@0"
I20260808 15:55:32.913947 2145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:32.950985 2145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:32.955227 2145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:32.955513 2145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:32.956451 2145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:32.957764 2145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:32.957841 2145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:32.958029 2145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:32.958079 2145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:32.984912 2005 heartbeater.cc:503] Master 127.27.254.62:44409 was elected leader, sending a full tablet report...
I20260808 15:55:32.991475 2145 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:39589
I20260808 15:55:32.991551 2272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:39589 every 8 connection(s)
I20260808 15:55:32.992810 2145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:32.998709 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2145
I20260808 15:55:32.998901 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:32.999493 28664 external_mini_cluster.cc:1428] Setting key 890cc297911696dc77fbefdad7d23345
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204532, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:33.012802 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:32.995390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:48931 (local address 127.27.254.62:44409)
0808 15:55:32.995892 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:32.995899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:32.996876 (+ 977us) server_negotiation.cc:407] Connection header received
0808 15:55:32.998903 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:32.998911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:32.999030 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:32.999139 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.001186 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.002190 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.003760 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.004175 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.004305 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.007804 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.007808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.007811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.007844 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.010306 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.010946 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.010950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.010952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.011066 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.011418 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.011421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.011423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.011663 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.011778 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.012383 (+ 605us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.012534 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260808 15:55:33.014133 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:32.996012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:44409 (local address 127.27.254.3:48931)
0808 15:55:32.996708 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:32.996753 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:32.998555 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:32.999423 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:32.999435 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:33.000168 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:33.001002 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:33.001037 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.002339 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.002348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:33.003614 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:33.003626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.004402 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.004409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:33.004635 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.005256 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:33.005278 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:33.007599 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:33.010424 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:33.010430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:33.010441 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:33.010851 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:33.011156 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:33.011159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:33.011161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:33.011328 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:33.011767 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:33.011773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:33.012067 (+ 294us) client_negotiation.cc:769] Sending connection context
0808 15:55:33.012308 (+ 241us) client_negotiation.cc:240] Negotiation successful
0808 15:55:33.012688 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":275,"threads_started":1}
I20260808 15:55:33.015506 2273 heartbeater.cc:347] Connected to a master server at 127.27.254.62:44409
I20260808 15:55:33.015818 2273 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:33.016505 2273 heartbeater.cc:511] Master 127.27.254.62:44409 requested a full tablet report, sending...
I20260808 15:55:33.017976 1816 ts_manager.cc:194] Registered new tserver with Master: 93ec382379d64fbe99c4218a7a860abf (127.27.254.3:39589)
I20260808 15:55:33.018854 1816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:48931
I20260808 15:55:33.027449 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/test-admin.keytab.
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.054293 28664 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 15:55:33.059463 2281 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:33.056919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.27.254.1:38357)
0808 15:55:33.057553 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.057562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.057597 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:55:33.057713 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.057720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.057931 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.058112 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.059128 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54972: BlockingRecv error: recv got EOF from 127.0.0.1:54972 (error 108)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":241,"threads_started":1}
W20260808 15:55:33.059777 28664 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.061360 28664 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.062745 28664 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:33.078112 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:33.064388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.27.254.1:38357)
0808 15:55:33.064631 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.064637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.064780 (+ 143us) server_negotiation.cc:407] Connection header received
0808 15:55:33.064995 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.064999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.065070 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.065179 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.066271 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.067267 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.068537 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.069155 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.069295 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.073006 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.073017 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.073032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.073079 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.075712 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.076273 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.076280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.076282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.076394 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.076707 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.076710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.076712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.077072 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.077288 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.077590 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.077776 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 15:55:33.083725 2281 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:33.082040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.27.254.1:38357)
0808 15:55:33.082212 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.082218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.082536 (+ 318us) server_negotiation.cc:407] Connection header received
0808 15:55:33.082716 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.082727 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.082777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.082872 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.083508 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54986: BlockingRecv error: recv got EOF from 127.0.0.1:54986 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 15:55:33.084100 28664 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.085649 28664 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.086939 28664 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:33.099946 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:33.088249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.27.254.1:38357)
0808 15:55:33.088438 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.088442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.088544 (+ 102us) server_negotiation.cc:407] Connection header received
0808 15:55:33.088667 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.088671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.088728 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.088802 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.089806 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.090492 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.091591 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.092033 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.092120 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.095525 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.095530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.095533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.095578 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.097874 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.098431 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.098435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.098437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.098541 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.098848 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.098850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.098852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.099090 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.099310 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.099543 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.099734 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:55:33.104224 2281 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:33.102497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.27.254.1:38357)
0808 15:55:33.102675 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.102682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.102968 (+ 286us) server_negotiation.cc:407] Connection header received
0808 15:55:33.103178 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.103181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.103244 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.103351 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.104011 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55006: BlockingRecv error: recv got EOF from 127.0.0.1:55006 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260808 15:55:33.104444 28664 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204485174398-28664-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.105952 28664 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:33.107343 28664 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:33.121074 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:33.108496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.27.254.1:38357)
0808 15:55:33.108680 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.108686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.108855 (+ 169us) server_negotiation.cc:407] Connection header received
0808 15:55:33.108961 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.108964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.109012 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.109087 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.110052 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.110724 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.111849 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.112263 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.112367 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.115818 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.115823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.115826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.115857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.118529 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.119158 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.119163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.119165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.119332 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.119704 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.119708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.119710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.119974 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.120199 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.120463 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.120701 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:55:33.122740 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1877
I20260808 15:55:33.134009 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:33.123410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.27.254.1:38357)
0808 15:55:33.123562 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.123566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.123962 (+ 396us) server_negotiation.cc:407] Connection header received
0808 15:55:33.124086 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.124089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.124141 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.124221 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.125060 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.125782 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.127050 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.127444 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.127535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.129376 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.129381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.129384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.129459 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.131822 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.132405 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.132408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.132411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.132533 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.132861 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.132864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.132866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.133107 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.133306 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.133614 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.133772 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:55:33.134578 2000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:33.217442 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1877
I20260808 15:55:33.234364 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2011
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:55:33.249221 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:33.235278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.27.254.2:42573)
0808 15:55:33.235871 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.235878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.235904 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:55:33.235966 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.235972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.236182 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.236311 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.237245 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.238309 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.239527 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.240056 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.240150 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.243933 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.243945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.243959 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.243999 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.246791 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.247477 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.247484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.247487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.247602 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.247904 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.247907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.247909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.248224 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.248450 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.248741 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.248933 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":325,"threads_started":1}
I20260808 15:55:33.250608 2134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:33.332727 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2011
I20260808 15:55:33.348933 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2145
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:55:33.364432 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:33.350045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.27.254.3:39589)
0808 15:55:33.350642 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:33.350649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:33.350677 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:55:33.350737 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:33.350743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:33.350951 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:33.351092 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:33.352069 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.353049 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.354274 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:33.354833 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:33.354931 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:33.358770 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:33.358780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:33.358794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:33.358837 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:33.361906 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.362552 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.362559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.362561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.362701 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:33.363048 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:33.363051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:33.363053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:33.363385 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:33.363629 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:33.363847 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 15:55:33.364120 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":299,"threads_started":1}
I20260808 15:55:33.365893 2268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:33.452812 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2145
I20260808 15:55:33.469839 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 1784
I20260808 15:55:33.471153 1846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:33.574162 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 1784
2026-08-08T15:55:33Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3930 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[2301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[2301](info): set up 2 sockets
Aug 08 15:55:33 dist-test-slave-srgv krb5kdc[2301](info): commencing operation
krb5kdc: starting...
W20260808 15:55:34.659907 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:34 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab.
Entry for principal oryx/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab.
I20260808 15:55:34.867174 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46835
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab
--principal=oryx/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:35.191810 2314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:35.192090 2314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:35.192245 2314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:35.199424 2314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:35.199523 2314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:35.199638 2314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:35.199744 2314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:35.199831 2314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:35.209599 2314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:46835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab
--principal=oryx/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:35.210929 2314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:35.212209 2314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:35.223670 2319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:35.223768 2320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:35.223769 2322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:35.224819 2314 server_base.cc:1034] running on GCE node
I20260808 15:55:35.225641 2314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:35.226955 2314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:35.228180 2314 hybrid_clock.cc:648] HybridClock initialized: now 1786204535228145 us; error 47 us; skew 500 ppm
I20260808 15:55:35.228511 2314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:35 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204535, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:35.231524 2314 init.cc:377] Logged in from keytab as oryx/127.27.254.62@KRBTEST.COM (short username oryx)
I20260808 15:55:35.232870 2314 webserver.cc:460] Webserver started at http://127.27.254.62:36765/ using document root <none> and password file <none>
I20260808 15:55:35.233588 2314 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:35.233739 2314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:35.234038 2314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:35.236877 2314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "993b6c092dd04040b740a1bfc66d0225"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "abe4333fff259299f289b6be812860a4"
server_key_iv: "42b15392436c2fb27a5c1e95b250dc1a"
server_key_version: "encryptionkey@0"
I20260808 15:55:35.237704 2314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "993b6c092dd04040b740a1bfc66d0225"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "abe4333fff259299f289b6be812860a4"
server_key_iv: "42b15392436c2fb27a5c1e95b250dc1a"
server_key_version: "encryptionkey@0"
I20260808 15:55:35.242735 2314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:55:35.246165 2329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:35.247428 2314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:55:35.247611 2314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "993b6c092dd04040b740a1bfc66d0225"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "abe4333fff259299f289b6be812860a4"
server_key_iv: "42b15392436c2fb27a5c1e95b250dc1a"
server_key_version: "encryptionkey@0"
I20260808 15:55:35.247821 2314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:35.263401 2314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:35.267839 2314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:35.268139 2314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:35.286571 2314 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:46835
I20260808 15:55:35.286602 2380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:46835 every 8 connection(s)
I20260808 15:55:35.287976 2314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:35.291538 2381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:35.297945 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2314
I20260808 15:55:35.298174 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:35.298822 28664 external_mini_cluster.cc:1428] Setting key 81ce1915d50fb8b3d8a39c94ab024a8e
I20260808 15:55:35.301826 2381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Bootstrap starting.
I20260808 15:55:35.305197 2381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:35.306406 2381 log.cc:824] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:35.309607 2381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: No bootstrap required, opened a new log
Aug 08 15:55:35 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:35.318377 2381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } }
I20260808 15:55:35.318728 2381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:35.318863 2381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b6c092dd04040b740a1bfc66d0225, State: Initialized, Role: FOLLOWER
I20260808 15:55:35.319510 2381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } }
I20260808 15:55:35.319782 2381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:35.319931 2381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:35.320112 2381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:35.322422 2384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:35.300797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.27.254.62:46835)
0808 15:55:35.301594 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:35.301605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:35.301646 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:55:35.303340 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:35.303363 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:35.303748 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:35.304053 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:35.305954 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:35.307723 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:35.309494 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:35.310214 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:35.310340 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:35.314966 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:35.314974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:35.314988 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:35.315025 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:35.318139 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:35.318884 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:35.318891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:35.318897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:35.319124 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:35.319518 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:35.319523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:35.319526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:35.320183 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:35.320509 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:35.320981 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 15:55:35.321402 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":343,"threads_started":1}
I20260808 15:55:35.322629 2381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } }
I20260808 15:55:35.323114 2381 leader_election.cc:302] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993b6c092dd04040b740a1bfc66d0225; no voters:
I20260808 15:55:35.323776 2381 leader_election.cc:288] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:35.323920 2386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:35.325852 2386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 LEADER]: Becoming Leader. State: Replica: 993b6c092dd04040b740a1bfc66d0225, State: Running, Role: LEADER
I20260808 15:55:35.326306 2381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:35.326444 2386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } }
I20260808 15:55:35.329134 2387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } }
I20260808 15:55:35.329505 2387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:35.329809 2388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993b6c092dd04040b740a1bfc66d0225. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } }
I20260808 15:55:35.330205 2388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:35.334127 2393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:35.338775 2393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:35.346876 2393 catalog_manager.cc:1347] Generated new cluster ID: ed6059a0fcae44d1bae514f36c3a0794
I20260808 15:55:35.346992 2393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:35.366797 2393 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:35.368069 2393 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:35.379788 2393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Generated new TSK 0
I20260808 15:55:35.380632 2393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab.
Entry for principal oryx/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab.
I20260808 15:55:35.437115 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46835
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:35.763307 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:35.763610 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:35.763783 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:35.771042 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:35.771214 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:35.771401 2408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:35.780973 2408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:35.782248 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:35.783566 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:35.796594 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:35.796592 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:35.796643 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:35.797397 2408 server_base.cc:1034] running on GCE node
I20260808 15:55:35.798103 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:35.799048 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:35.800267 2408 hybrid_clock.cc:648] HybridClock initialized: now 1786204535800215 us; error 51 us; skew 500 ppm
I20260808 15:55:35.800582 2408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:35 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204535, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:35.803714 2408 init.cc:377] Logged in from keytab as oryx/127.27.254.1@KRBTEST.COM (short username oryx)
I20260808 15:55:35.805120 2408 webserver.cc:460] Webserver started at http://127.27.254.1:34807/ using document root <none> and password file <none>
I20260808 15:55:35.805859 2408 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:35.806006 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:35.806310 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:35.809118 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "d72fbc8141f448ee82771422f4411e37"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "9ed2a22cb38086765cb4dd73af6a3a5b"
server_key_iv: "910b68c734d955fb531d226ccd4e6606"
server_key_version: "encryptionkey@0"
I20260808 15:55:35.809908 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "d72fbc8141f448ee82771422f4411e37"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "9ed2a22cb38086765cb4dd73af6a3a5b"
server_key_iv: "910b68c734d955fb531d226ccd4e6606"
server_key_version: "encryptionkey@0"
I20260808 15:55:35.814922 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:55:35.818394 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:35.819833 2408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:55:35.820007 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "d72fbc8141f448ee82771422f4411e37"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "9ed2a22cb38086765cb4dd73af6a3a5b"
server_key_iv: "910b68c734d955fb531d226ccd4e6606"
server_key_version: "encryptionkey@0"
I20260808 15:55:35.820199 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:35.849635 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:35.854190 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:35.854490 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:35.855427 2408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:35.856792 2408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:35.856863 2408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:35.857024 2408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:35.857084 2408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:35.888798 2408 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:40339
I20260808 15:55:35.888846 2535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:40339 every 8 connection(s)
I20260808 15:55:35.890245 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:35.899809 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2408
I20260808 15:55:35.900020 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:35.900877 28664 external_mini_cluster.cc:1428] Setting key b4f8880699aaac5c769ef75985401071
Aug 08 15:55:35 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204535, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:35.910985 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:35.892846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:38341 (local address 127.27.254.62:46835)
0808 15:55:35.893239 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:35.893243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:35.894298 (+ 1055us) server_negotiation.cc:407] Connection header received
0808 15:55:35.896024 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:35.896028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:35.896093 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:35.896195 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:35.897627 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:35.898295 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:35.899819 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:35.900490 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:35.900633 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:35.904229 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:35.904239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:35.904248 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:35.904294 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:35.907744 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:35.908431 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:35.908439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:35.908446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:35.908597 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:35.908987 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:35.908995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:35.909001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:35.909390 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:35.909574 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:35.910285 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 15:55:35.910535 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":195,"threads_started":1}
I20260808 15:55:35.911561 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:35.893476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:38341)
0808 15:55:35.894139 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:35.894186 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:35.895766 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:35.896396 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:35.896404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:35.896883 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:35.897469 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:35.897496 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:35.898420 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:35.898423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:35.899667 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:35.899678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:35.900698 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:35.900703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:35.900896 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:35.901636 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:35.901658 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:35.903995 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:35.907887 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:35.907893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:35.907905 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:35.908300 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:35.908700 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:35.908703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:35.908706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:35.908871 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:35.909545 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:35.909550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:35.909879 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 15:55:35.910104 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 15:55:35.910469 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":258,"threads_started":1}
I20260808 15:55:35.913079 2536 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46835
I20260808 15:55:35.913393 2536 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:35.914130 2536 heartbeater.cc:511] Master 127.27.254.62:46835 requested a full tablet report, sending...
I20260808 15:55:35.916198 2346 ts_manager.cc:194] Registered new tserver with Master: d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339)
I20260808 15:55:35.917685 2346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:38341
WARNING: no policy specified for oryx/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab.
Entry for principal oryx/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab.
I20260808 15:55:35.948954 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46835
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:36.265252 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:36.265543 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:36.265708 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:36.272850 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:36.272953 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:36.273172 2542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:36.282667 2542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:36.283953 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:36.285259 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:36.299762 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:36.299860 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:36.299911 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:36.301435 2542 server_base.cc:1034] running on GCE node
I20260808 15:55:36.302131 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:36.303290 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:36.304523 2542 hybrid_clock.cc:648] HybridClock initialized: now 1786204536304459 us; error 56 us; skew 500 ppm
I20260808 15:55:36.304893 2542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:36 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204536, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:36.311576 2542 init.cc:377] Logged in from keytab as oryx/127.27.254.2@KRBTEST.COM (short username oryx)
I20260808 15:55:36.313012 2542 webserver.cc:460] Webserver started at http://127.27.254.2:43839/ using document root <none> and password file <none>
I20260808 15:55:36.313903 2542 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:36.314031 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:36.314338 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:36.317238 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "620e184eeaf44ad19d7c4dd6140af409"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "09fcfdb607f345da6e115eddae5aafb6"
server_key_iv: "5b946405a4a75b4b071983e2d491d465"
server_key_version: "encryptionkey@0"
I20260808 15:55:36.318065 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "620e184eeaf44ad19d7c4dd6140af409"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "09fcfdb607f345da6e115eddae5aafb6"
server_key_iv: "5b946405a4a75b4b071983e2d491d465"
server_key_version: "encryptionkey@0"
I20260808 15:55:36.322932 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:55:36.326242 2557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:36.327617 2542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:55:36.327796 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "620e184eeaf44ad19d7c4dd6140af409"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "09fcfdb607f345da6e115eddae5aafb6"
server_key_iv: "5b946405a4a75b4b071983e2d491d465"
server_key_version: "encryptionkey@0"
I20260808 15:55:36.327973 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:36.357779 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:36.362183 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:36.362483 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:36.363481 2542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:36.364825 2542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:36.364895 2542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:36.365041 2542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:36.365108 2542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:36.398695 2542 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:35825
I20260808 15:55:36.398782 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:35825 every 8 connection(s)
I20260808 15:55:36.399993 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:36.409528 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2542
I20260808 15:55:36.409715 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:36.410295 28664 external_mini_cluster.cc:1428] Setting key 23d6d79c2dd96ff0443b74f78470859c
Aug 08 15:55:36 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204536, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:36.420276 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:36.402574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:59003 (local address 127.27.254.62:46835)
0808 15:55:36.402789 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:36.402795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:36.403938 (+ 1143us) server_negotiation.cc:407] Connection header received
0808 15:55:36.405705 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:36.405709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:36.405813 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:36.405908 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:36.407271 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.407957 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.409431 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.410134 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.410267 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:36.413759 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:36.413764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:36.413767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:36.413796 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:36.416971 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:36.417965 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:36.417973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:36.417978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:36.418119 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:36.418640 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:36.418645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:36.418648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:36.418911 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:36.419082 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:36.419868 (+ 786us) server_negotiation.cc:299] Negotiation successful
0808 15:55:36.420034 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:55:36.422293 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:36.403098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:59003)
0808 15:55:36.403756 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:36.403817 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:36.405401 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:36.406107 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:36.406114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:36.406588 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:36.407127 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:36.407145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.408080 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.408084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:36.409265 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:36.409277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.410328 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.410333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:36.410587 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:36.411295 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:36.411318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:36.413534 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:36.417138 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:36.417149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:36.417168 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:36.417851 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:36.418286 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:36.418294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:36.418300 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:36.418505 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:36.419041 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:36.419049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:36.419477 (+ 428us) client_negotiation.cc:769] Sending connection context
0808 15:55:36.419774 (+ 297us) client_negotiation.cc:240] Negotiation successful
0808 15:55:36.420344 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":269,"threads_started":1}
I20260808 15:55:36.423820 2670 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46835
I20260808 15:55:36.424160 2670 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:36.424885 2670 heartbeater.cc:511] Master 127.27.254.62:46835 requested a full tablet report, sending...
I20260808 15:55:36.426483 2346 ts_manager.cc:194] Registered new tserver with Master: 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825)
I20260808 15:55:36.427804 2346 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:59003
WARNING: no policy specified for oryx/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab.
Entry for principal oryx/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab.
I20260808 15:55:36.458416 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46835
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:36.777202 2675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:36.777521 2675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:36.777709 2675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:36.784629 2675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:36.784768 2675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:36.784948 2675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:36.794086 2675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:36.795414 2675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:36.796701 2675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:36.809724 2683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:36.809746 2680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:36.809849 2681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:36.810276 2675 server_base.cc:1034] running on GCE node
I20260808 15:55:36.810963 2675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:36.811865 2675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:36.813094 2675 hybrid_clock.cc:648] HybridClock initialized: now 1786204536813069 us; error 35 us; skew 500 ppm
I20260808 15:55:36.813385 2675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:36 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204536, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:36.816382 2675 init.cc:377] Logged in from keytab as oryx/127.27.254.3@KRBTEST.COM (short username oryx)
I20260808 15:55:36.817715 2675 webserver.cc:460] Webserver started at http://127.27.254.3:36987/ using document root <none> and password file <none>
I20260808 15:55:36.818312 2675 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:36.818447 2675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:36.818722 2675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:36.821481 2675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "e9f32ff8506f4806ab40be69e08ec0d0"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "47ebf0bd2a6e80c847f3090d25395488"
server_key_iv: "fdfddfe79907cf1eb01793b496fd950c"
server_key_version: "encryptionkey@0"
I20260808 15:55:36.822191 2675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "e9f32ff8506f4806ab40be69e08ec0d0"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "47ebf0bd2a6e80c847f3090d25395488"
server_key_iv: "fdfddfe79907cf1eb01793b496fd950c"
server_key_version: "encryptionkey@0"
I20260808 15:55:36.826956 2675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:55:36.830231 2690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:36.831472 2675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 15:55:36.831657 2675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "e9f32ff8506f4806ab40be69e08ec0d0"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "47ebf0bd2a6e80c847f3090d25395488"
server_key_iv: "fdfddfe79907cf1eb01793b496fd950c"
server_key_version: "encryptionkey@0"
I20260808 15:55:36.831862 2675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:36.856423 2675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:36.860158 2675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:36.860450 2675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:36.861387 2675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:36.862722 2675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:36.862828 2675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:36.862978 2675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:36.863054 2675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:36.896237 2675 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:43015
I20260808 15:55:36.896265 2804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:43015 every 8 connection(s)
I20260808 15:55:36.897481 2675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:36.907289 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2675
I20260808 15:55:36.907476 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:36.907940 28664 external_mini_cluster.cc:1428] Setting key 6dc1da970044aae26dd923270f137ea2
Aug 08 15:55:36 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204536, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:36.916859 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:36.899959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:43967 (local address 127.27.254.62:46835)
0808 15:55:36.900150 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:36.900155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:36.901234 (+ 1079us) server_negotiation.cc:407] Connection header received
0808 15:55:36.903088 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:36.903094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:36.903203 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:36.903318 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:36.904738 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.905583 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.907131 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.907664 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.907799 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:36.911685 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:36.911690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:36.911693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:36.911725 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:36.914188 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:36.914828 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:36.914832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:36.914834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:36.914952 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:36.915409 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:36.915413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:36.915415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:36.915691 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:36.915825 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:36.916435 (+ 610us) server_negotiation.cc:299] Negotiation successful
0808 15:55:36.916614 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:55:36.917855 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:36.900421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:43967)
0808 15:55:36.901080 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:36.901116 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:36.902808 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:36.903550 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:36.903558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:36.904032 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:36.904569 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:36.904585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.905735 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.905742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:36.906980 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:36.906991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:36.907898 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:36.907906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:36.908219 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:36.908912 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:36.908942 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:36.911512 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:36.914305 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:36.914311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:36.914323 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:36.914729 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:36.915044 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:36.915068 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:36.915074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:36.915269 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:36.915815 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:36.915821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:36.916104 (+ 283us) client_negotiation.cc:769] Sending connection context
0808 15:55:36.916333 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 15:55:36.916687 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":258,"threads_started":1}
I20260808 15:55:36.919308 2805 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46835
I20260808 15:55:36.919601 2805 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:36.920375 2805 heartbeater.cc:511] Master 127.27.254.62:46835 requested a full tablet report, sending...
I20260808 15:55:36.920445 2536 heartbeater.cc:503] Master 127.27.254.62:46835 was elected leader, sending a full tablet report...
I20260808 15:55:36.921940 2345 ts_manager.cc:194] Registered new tserver with Master: e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015)
I20260808 15:55:36.923120 2345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:43967
I20260808 15:55:36.923339 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.61.keytab.
Entry for principal oryx/127.27.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.61.keytab.
I20260808 15:55:36.970863 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:35737
--webserver_interface=127.27.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.61.keytab
--principal=oryx/127.27.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:46835,127.27.254.61:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:37.291082 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:37.291415 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:37.291580 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:37.298617 2810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:37.298758 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:37.298862 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:37.298964 2810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:37.299031 2810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:37.308444 2810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:46835,127.27.254.61:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.61.keytab
--principal=oryx/127.27.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
W20260808 15:55:37.308842 2810 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.254.62:46835,127.27.254.61:35737'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 15:55:37.309921 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:37.311267 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:37.322855 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:37.322850 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:37.322883 2818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:37.323761 2810 server_base.cc:1034] running on GCE node
I20260808 15:55:37.324486 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:37.325805 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:37.327010 2810 hybrid_clock.cc:648] HybridClock initialized: now 1786204537326959 us; error 33 us; skew 500 ppm
I20260808 15:55:37.327442 2810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:37.330629 2810 init.cc:377] Logged in from keytab as oryx/127.27.254.61@KRBTEST.COM (short username oryx)
I20260808 15:55:37.332170 2810 webserver.cc:460] Webserver started at http://127.27.254.61:36885/ using document root <none> and password file <none>
I20260808 15:55:37.332832 2810 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:37.332916 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:37.333165 2810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:37.336149 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/instance:
uuid: "a66c7b522c5e45a096157f6b545f2143"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "36e6e30024260b1fd0c20a0c57082964"
server_key_iv: "9eda2002ac77d38231445271f5b37977"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.336934 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal/instance:
uuid: "a66c7b522c5e45a096157f6b545f2143"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "36e6e30024260b1fd0c20a0c57082964"
server_key_iv: "9eda2002ac77d38231445271f5b37977"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.341941 2810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:55:37.345134 2825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:37.346649 2810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:37.346833 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
uuid: "a66c7b522c5e45a096157f6b545f2143"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "36e6e30024260b1fd0c20a0c57082964"
server_key_iv: "9eda2002ac77d38231445271f5b37977"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.347048 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:37.369493 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:37.373781 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:37.374050 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:37.392264 2810 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.61:35737
I20260808 15:55:37.392288 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.61:35737 every 8 connection(s)
I20260808 15:55:37.393642 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
I20260808 15:55:37.397360 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:37.401018 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2810
I20260808 15:55:37.405441 2878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.61@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:37.424508 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.408345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.27.254.62:46835)
0808 15:55:37.408492 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.408497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.409087 (+ 590us) server_negotiation.cc:407] Connection header received
0808 15:55:37.409237 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.409240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.409299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.409380 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.410774 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.411921 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.413817 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.414460 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.414607 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.418448 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.418453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.418456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.418487 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.421279 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.422064 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.422075 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.422078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.422221 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.422630 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.422634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.422636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.423087 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.423261 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.423910 (+ 649us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.424183 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 15:55:37.425091 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.408239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54434)
0808 15:55:37.408917 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:37.408958 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:37.409119 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:37.409483 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:37.409490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:37.410313 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:37.410631 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.410639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.412056 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.412059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.413671 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.413680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.414698 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.414703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.414826 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.415491 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:37.415511 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:37.418254 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:37.421459 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.421470 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.421477 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.421932 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.422343 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.422349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.422353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.422499 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.423223 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:37.423232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:37.423513 (+ 281us) client_negotiation.cc:769] Sending connection context
0808 15:55:37.423753 (+ 240us) client_negotiation.cc:240] Negotiation successful
0808 15:55:37.423997 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":279,"threads_started":1}
I20260808 15:55:37.425109 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.403476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.27.254.61:35737)
0808 15:55:37.404344 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.404360 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.404410 (+ 50us) server_negotiation.cc:407] Connection header received
0808 15:55:37.406036 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.406056 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.406438 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.406795 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.408302 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.410357 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.412210 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.413182 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.413374 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.417959 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.417967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.417979 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.418006 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.420689 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.421383 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.421393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.421442 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.421642 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.422023 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.422030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.422035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.422541 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.422867 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.423378 (+ 511us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.423829 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":370,"threads_started":1}
I20260808 15:55:37.427146 2878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:55:37.430390 2670 heartbeater.cc:503] Master 127.27.254.62:46835 was elected leader, sending a full tablet report...
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.61@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
I20260808 15:55:37.439615 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.427949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41360)
0808 15:55:37.428173 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:37.428192 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:37.428297 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:37.428593 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:37.428597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:37.428944 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:37.429057 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.429065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.430084 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.430092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.431044 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.431055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.431761 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.431766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.432040 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.432559 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:37.432575 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:37.434482 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:37.437272 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.437278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.437282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.437700 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.438042 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.438046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.438050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.438178 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.438654 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:37.438658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:37.438806 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 15:55:37.438938 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 15:55:37.439237 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":22}
I20260808 15:55:37.439646 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.428217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.27.254.61:35737)
0808 15:55:37.428367 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.428374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.428389 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:55:37.428428 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.428431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.428472 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.428561 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.429179 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.429942 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.431165 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.431624 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.431754 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.434657 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.434661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.434665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.434689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.437128 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.437810 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.437818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.437822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.437936 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.438276 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.438284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.438287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.438552 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.438690 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.439012 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.439362 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":212,"server-negotiator.queue_time_us":49}
I20260808 15:55:37.444358 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Bootstrap starting.
I20260808 15:55:37.447366 2878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:37.448312 2878 log.cc:824] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:37.450948 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: No bootstrap required, opened a new log
I20260808 15:55:37.458084 2878 raft_consensus.cc:348] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } }
I20260808 15:55:37.458420 2878 raft_consensus.cc:374] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:37.458521 2878 raft_consensus.cc:729] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66c7b522c5e45a096157f6b545f2143, State: Initialized, Role: FOLLOWER
I20260808 15:55:37.459095 2878 consensus_queue.cc:260] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } }
I20260808 15:55:37.459856 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } } }
I20260808 15:55:37.460132 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:55:37.460872 2878 sys_catalog.cc:564] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:55:37.469111 2895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:55:37.469251 2895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:55:37.471835 2810 master_runner.cc:428] Detected that this master a66c7b522c5e45a096157f6b545f2143 is joining an existing cluster
I20260808 15:55:37.471953 2810 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.254.61:35737
I20260808 15:55:37.493999 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.475765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54448)
0808 15:55:37.477983 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:37.478009 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:37.478227 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:37.479031 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:37.479036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:37.479847 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:37.480268 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.480280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.481791 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.481796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.483550 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.483564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.484800 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.484806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.484989 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.485792 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:37.485817 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:37.487045 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:37.491131 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.491136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.491139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.491708 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.492139 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.492144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.492151 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.492311 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.492849 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:37.492854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:37.493033 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 15:55:37.493215 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 15:55:37.493536 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2024,"mutex_wait_us":35,"spinlock_wait_cycles":31104,"thread_start_us":1841,"threads_started":1}
I20260808 15:55:37.493999 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.475696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.27.254.62:46835)
0808 15:55:37.475898 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.475904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.478112 (+ 2208us) server_negotiation.cc:407] Connection header received
0808 15:55:37.478648 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.478654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.478742 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.478852 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.480452 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.481586 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.483752 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.484405 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.484503 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.487267 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.487274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.487278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.487316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.490991 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.491841 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.491846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.491850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.492019 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.492434 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.492442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.492447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.492735 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.492847 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.493359 (+ 512us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.493634 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:55:37.495296 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.476296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.27.254.61:35737)
0808 15:55:37.476527 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.476535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.478439 (+ 1904us) server_negotiation.cc:407] Connection header received
0808 15:55:37.478587 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.478592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.478664 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.478755 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.480324 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.481492 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.483819 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.484604 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.484711 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.487668 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.487677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.487684 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.487725 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.491735 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.492894 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.492897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.492899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.492995 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.493404 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.493408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.493411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.493776 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.493909 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.494777 (+ 868us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.494979 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:55:37.496335 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.475863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41372)
0808 15:55:37.478338 (+ 2475us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:37.478355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:37.478473 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:37.478934 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:37.478938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:37.479696 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:37.480158 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.480171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.481703 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.481710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.483667 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.483686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.485013 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.485020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.485608 (+ 588us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.486244 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:37.486270 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:37.487489 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:37.491889 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.491895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.491901 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.492327 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.493109 (+ 782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.493117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.493120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.493291 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.494008 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:37.494015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:37.494227 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 15:55:37.495643 (+ 1416us) client_negotiation.cc:240] Negotiation successful
0808 15:55:37.495912 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2322,"mutex_wait_us":71,"thread_start_us":2148,"threads_started":1}
WARNING: no policy specified for oryx/127.27.254.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.60@KRBTEST.COM" created.
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204535, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.62@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
I20260808 15:55:37.518595 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.501738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.27.254.61:35737)
0808 15:55:37.501909 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.501915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.502394 (+ 479us) server_negotiation.cc:407] Connection header received
0808 15:55:37.502679 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.502684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.502779 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.502877 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.504403 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.505266 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.507231 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.508080 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.508223 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.512583 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.512588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.512591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.512622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.515517 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.516225 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.516237 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.516240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.516373 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.516927 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.516933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.516937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.517282 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.517441 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.517977 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.518201 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:55:37.518646 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.501621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41384)
0808 15:55:37.502253 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:37.502296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:37.502457 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:37.503102 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:37.503109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:37.503951 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:37.504215 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.504228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.505399 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.505442 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.507041 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.507057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.508265 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.508272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.508488 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.509228 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:37.509258 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:37.512372 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:37.515657 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.515670 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.515678 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.516090 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.516574 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.516581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.516586 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.516777 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.517523 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:37.517538 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:37.517690 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 15:55:37.517875 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 15:55:37.518222 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":278,"threads_started":1}
I20260808 15:55:37.520790 2345 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.254.61:35737: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a66c7b522c5e45a096157f6b545f2143"
member_type: NON_VOTER
last_known_addr {
host: "127.27.254.61"
port: 35737
}
attrs {
promote: true
}
}
dest_uuid: "993b6c092dd04040b740a1bfc66d0225"
cas_config_opid_index: -1
I20260808 15:55:37.522133 2345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } }
I20260808 15:55:37.526680 2908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a66c7b522c5e45a096157f6b545f2143 (127.27.254.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } }
I20260808 15:55:37.527827 2852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:37.530995 2908 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.254.61:35737
I20260808 15:55:37.532007 2810 master.cc:537] Master@127.27.254.61:35737 shutting down...
I20260808 15:55:37.532178 2852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Refusing update from remote peer 993b6c092dd04040b740a1bfc66d0225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 15:55:37.533322 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } }
I20260808 15:55:37.533713 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:37.534963 2908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:55:37.538867 2330 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 15:55:37.540393 2330 consensus_peers.cc:487] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 -> Peer a66c7b522c5e45a096157f6b545f2143 (127.27.254.61:35737): Couldn't send request to peer a66c7b522c5e45a096157f6b545f2143. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.60.keytab.
Entry for principal oryx/127.27.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.60.keytab.
I20260808 15:55:37.543257 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:43385
--webserver_interface=127.27.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.60.keytab
--principal=oryx/127.27.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:55:37.550861 2810 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:37.551347 2810 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:37.551596 2810 tablet_replica.cc:330] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: stopping tablet replica
I20260808 15:55:37.557925 2810 master.cc:552] Master@127.27.254.61:35737 shutdown complete.
I20260808 15:55:37.558076 2810 master_runner.cc:305] Clearing existing system tablet
I20260808 15:55:37.558506 2810 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:37.558620 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:37.560918 2810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 15:55:37.562644 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:37.562965 2810 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:37.563282 2810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 15:55:37.563390 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
uuid: "a66c7b522c5e45a096157f6b545f2143"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "36e6e30024260b1fd0c20a0c57082964"
server_key_iv: "9eda2002ac77d38231445271f5b37977"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.566896 2810 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:37.571575 2810 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 15:55:37.571774 2810 log.cc:1196] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 15:55:37.572297 2810 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Deleting consensus metadata
I20260808 15:55:37.572552 2810 master_runner.cc:315] Copying system tablet from 127.27.254.62:46835
I20260808 15:55:37.575181 2810 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: tablet copy: Beginning tablet copy session from remote peer at address 127.27.254.62:46835
I20260808 15:55:37.587662 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.575992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.27.254.62:46835)
0808 15:55:37.576208 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:37.576218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:37.576711 (+ 493us) server_negotiation.cc:407] Connection header received
0808 15:55:37.576914 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:37.576920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:37.577010 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:37.577127 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:37.578042 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.578767 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.580114 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.580525 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.580670 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.582435 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:37.582448 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:37.582451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:37.582481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:37.585052 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.585755 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.585762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.585765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.585890 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:37.586260 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:37.586265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:37.586269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:37.586555 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:37.586732 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:37.587100 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 15:55:37.587318 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:55:37.587714 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:37.575962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54460)
0808 15:55:37.576593 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:37.576609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:37.576779 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:37.577151 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:37.577156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:37.577636 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:37.577853 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.577862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.578902 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.578910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.579962 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:37.579974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:37.580680 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:37.580685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:37.580860 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:37.581530 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:37.581547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:37.582220 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:37.585196 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.585202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.585206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.585641 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.586002 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:37.586007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:37.586011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:37.586140 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:37.586671 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:37.586678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:37.586853 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 15:55:37.586985 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 15:55:37.587268 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":396,"threads_started":1}
I20260808 15:55:37.588840 2366 tablet_copy_service.cc:140] P 993b6c092dd04040b740a1bfc66d0225: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a66c7b522c5e45a096157f6b545f2143 ({username='oryx', principal='oryx/127.27.254.61@KRBTEST.COM'} at 127.0.0.1:54460)
I20260808 15:55:37.589165 2366 tablet_copy_service.cc:161] P 993b6c092dd04040b740a1bfc66d0225: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a66c7b522c5e45a096157f6b545f2143 at {username='oryx', principal='oryx/127.27.254.61@KRBTEST.COM'} at 127.0.0.1:54460: session id = a66c7b522c5e45a096157f6b545f2143-00000000000000000000000000000000
I20260808 15:55:37.592540 2366 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 15:55:37.594748 2810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 15:55:37.599256 2810 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: tablet copy: Starting download of 0 data blocks...
I20260808 15:55:37.599685 2810 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: tablet copy: Starting download of 1 WAL segments...
I20260808 15:55:37.602314 2810 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 15:55:37.605603 2366 tablet_copy_service.cc:342] P 993b6c092dd04040b740a1bfc66d0225: Request end of tablet copy session a66c7b522c5e45a096157f6b545f2143-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.254.61@KRBTEST.COM'} at 127.0.0.1:54460
I20260808 15:55:37.605839 2366 tablet_copy_service.cc:433] P 993b6c092dd04040b740a1bfc66d0225: ending tablet copy session a66c7b522c5e45a096157f6b545f2143-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a66c7b522c5e45a096157f6b545f2143
I20260808 15:55:37.617314 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:37.620023 2920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:37.620479 2921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:37.627918 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:37.628345 2810 server_base.cc:1034] running on GCE node
I20260808 15:55:37.628841 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:37.629561 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:37.630860 2810 hybrid_clock.cc:648] HybridClock initialized: now 1786204537630781 us; error 35 us; skew 500 ppm
I20260808 15:55:37.631213 2810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:37.633738 2810 init.cc:377] Logged in from keytab as oryx/127.27.254.61@KRBTEST.COM (short username oryx)
I20260808 15:55:37.638341 2810 webserver.cc:460] Webserver started at http://127.27.254.61:42005/ using document root <none> and password file <none>
I20260808 15:55:37.638790 2810 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:37.638934 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:37.642275 2810 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 15:55:37.644496 2930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:37.644986 2810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 15:55:37.645219 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
uuid: "a66c7b522c5e45a096157f6b545f2143"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "36e6e30024260b1fd0c20a0c57082964"
server_key_iv: "9eda2002ac77d38231445271f5b37977"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.645546 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:37.675814 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:37.679936 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:37.702411 2810 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.61:35737
I20260808 15:55:37.702453 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.61:35737 every 8 connection(s)
I20260808 15:55:37.703610 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
I20260808 15:55:37.708849 2982 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:55:37.712707 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Bootstrap starting.
I20260808 15:55:37.728981 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:55:37.729542 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Bootstrap complete.
I20260808 15:55:37.730611 2982 raft_consensus.cc:348] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } }
I20260808 15:55:37.730871 2982 raft_consensus.cc:729] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a66c7b522c5e45a096157f6b545f2143, State: Initialized, Role: LEARNER
I20260808 15:55:37.731101 2982 consensus_queue.cc:260] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } }
I20260808 15:55:37.732323 2982 sys_catalog.cc:564] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:37.732085 2985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } }
I20260808 15:55:37.732715 2985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:55:37.740602 2996 catalog_manager.cc:1259] Loaded cluster ID: ed6059a0fcae44d1bae514f36c3a0794
I20260808 15:55:37.740710 2996 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: loading cluster ID for follower catalog manager: success
I20260808 15:55:37.744477 2996 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: acquiring CA information for follower catalog manager: success
I20260808 15:55:37.747022 2996 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:55:37.925619 2805 heartbeater.cc:503] Master 127.27.254.62:46835 was elected leader, sending a full tablet report...
W20260808 15:55:37.931843 2911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:37.932265 2911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:37.932426 2911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:37.940495 2911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:37.940640 2911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:37.940750 2911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:37.940852 2911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:37.940928 2911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:37.954834 2911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.60.keytab
--principal=oryx/127.27.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:37.956270 2911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:37.957674 2911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:37.969272 3001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:37.969213 3002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:37.969509 3004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:37.970304 2911 server_base.cc:1034] running on GCE node
I20260808 15:55:37.971035 2911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:37.972347 2911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:37.973559 2911 hybrid_clock.cc:648] HybridClock initialized: now 1786204537973512 us; error 56 us; skew 500 ppm
I20260808 15:55:37.973912 2911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:37 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:37.976974 2911 init.cc:377] Logged in from keytab as oryx/127.27.254.60@KRBTEST.COM (short username oryx)
I20260808 15:55:37.978421 2911 webserver.cc:460] Webserver started at http://127.27.254.60:43893/ using document root <none> and password file <none>
I20260808 15:55:37.979063 2911 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:37.979171 2911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:37.979414 2911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:37.982372 2911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/instance:
uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "4e804b57da788421a109c1d4c7d40b15"
server_key_iv: "785c8bb1ef86660e9d6125cf03f80e83"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.983073 2911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal/instance:
uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "4e804b57da788421a109c1d4c7d40b15"
server_key_iv: "785c8bb1ef86660e9d6125cf03f80e83"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.988091 2911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:55:37.991477 3012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:37.992820 2911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:37.992969 2911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "4e804b57da788421a109c1d4c7d40b15"
server_key_iv: "785c8bb1ef86660e9d6125cf03f80e83"
server_key_version: "encryptionkey@0"
I20260808 15:55:37.993139 2911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:38.016448 2911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:38.020787 2911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:38.021029 2911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:38.042320 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.033969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41386)
0808 15:55:38.034808 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.034847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.034979 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.036254 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.036271 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.036302 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:38.036626 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.036642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.038407 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.038414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.040358 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.040372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.041526 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.041533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.041700 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.041763 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.041883 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.042055 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":615,"threads_started":1}
I20260808 15:55:38.042608 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.034144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.27.254.61:35737)
0808 15:55:38.035848 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.035853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.035870 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:38.035953 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.035958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.036086 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.036215 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:38.036818 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.038271 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.040519 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.041315 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.041508 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.041738 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.041947 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.042144 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"thread_start_us":1487,"threads_started":1}
I20260808 15:55:38.043752 2957 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260808 15:55:38.044358 2985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993b6c092dd04040b740a1bfc66d0225. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } }
I20260808 15:55:38.044629 2985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:55:38.045861 2911 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.60:43385
I20260808 15:55:38.045867 3065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.60:43385 every 8 connection(s)
I20260808 15:55:38.047362 2911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
I20260808 15:55:38.048454 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 2911
I20260808 15:55:38.052143 3066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:38.059798 3066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.60@KRBTEST.COM
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.60@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:38.077224 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.062841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.27.254.62:46835)
0808 15:55:38.063046 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.063054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.063532 (+ 478us) server_negotiation.cc:407] Connection header received
0808 15:55:38.063681 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.063685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.063748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.063828 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.064919 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.065832 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.067285 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.068023 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.068174 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.071625 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.071630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.071634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.071664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.074455 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.075178 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.075183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.075188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.075314 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.075696 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.075702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.075706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.075960 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.076130 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.076613 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.076846 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:55:38.078070 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.062588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54472)
0808 15:55:38.063360 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.063409 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.063561 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.063924 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.063934 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.064525 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.064757 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.064765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.065967 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.065972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.067135 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.067147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.068225 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.068230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.068346 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.068968 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.068988 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.071436 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.074594 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.074607 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.074626 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.075051 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.075425 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.075432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.075436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.075582 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.076092 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.076100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.076289 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.076522 (+ 233us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.077091 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":352,"threads_started":1}
I20260808 15:55:38.078095 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.050812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.27.254.60:43385)
0808 15:55:38.051818 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.051842 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.057031 (+ 5189us) server_negotiation.cc:407] Connection header received
0808 15:55:38.059407 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.059436 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.059987 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.060434 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.061899 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.063849 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.065246 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.065964 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.066161 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.071197 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.071205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.071216 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.071243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.073920 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.074618 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.074626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.074628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.074831 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.075244 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.075248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.075250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.075744 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.076041 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.076474 (+ 433us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.076862 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"spinlock_wait_cycles":27008,"thread_start_us":375,"threads_started":1}
I20260808 15:55:38.080492 3066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.60@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
I20260808 15:55:38.092432 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.081271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41394)
0808 15:55:38.081486 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.081506 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.081595 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.081848 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.081851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.082153 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.082277 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.082284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.083197 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.083200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.084074 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.084083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.084708 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.084712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.084840 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.085442 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.085472 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.087393 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.090406 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.090410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.090413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.090796 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.091147 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.091151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.091154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.091285 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.091739 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.091742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.091854 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.091962 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.092178 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:55:38.092471 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.081437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.27.254.61:35737)
0808 15:55:38.081589 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.081594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.081610 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:55:38.081681 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.081685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.081740 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.081816 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.082399 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.083087 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.084194 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.084589 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.084681 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.087551 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.087555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.087559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.087588 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.090272 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.090897 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.090900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.090903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.091020 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.091380 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.091383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.091385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.091628 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.091749 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.092041 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.092220 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 15:55:38.093739 3066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.60@KRBTEST.COM for oryx/127.27.254.60@KRBTEST.COM
I20260808 15:55:38.105629 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.094367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.0.0.1:41096)
0808 15:55:38.094534 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.094558 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.094649 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.094920 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.094923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.095229 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.095365 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.095372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.096331 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.096334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.097470 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.097485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.098193 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.098197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.098373 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.098881 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.098897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.100773 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.103539 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.103543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.103545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.103887 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.104223 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.104226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.104229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.104381 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.104815 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.104818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.104929 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.105058 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.105258 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:55:38.105659 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.094563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.27.254.60:43385)
0808 15:55:38.094709 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.094713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.094727 (+ 14us) server_negotiation.cc:407] Connection header received
0808 15:55:38.094763 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.094766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.094813 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.094888 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.095516 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.096215 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.097596 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.098070 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.098177 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.100929 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.100934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.100937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.100964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.103428 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.103979 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.103982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.103984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.104083 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.104475 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.104478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.104480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.104719 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.104822 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.105136 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.105361 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":51}
I20260808 15:55:38.110339 3066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Bootstrap starting.
I20260808 15:55:38.113364 3066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:38.114415 3066 log.cc:824] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:38.116979 3066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: No bootstrap required, opened a new log
I20260808 15:55:38.123968 3066 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } }
I20260808 15:55:38.124316 3066 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:38.124430 3066 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3259a5e0d32465faba0cfb4fe1e919f, State: Initialized, Role: FOLLOWER
I20260808 15:55:38.125002 3066 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } }
I20260808 15:55:38.125919 3072 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } } }
I20260808 15:55:38.126341 3072 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:55:38.127054 3066 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:38.133438 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2408
W20260808 15:55:38.136549 3083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:55:38.136699 3083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:55:38.143723 2911 master_runner.cc:428] Detected that this master c3259a5e0d32465faba0cfb4fe1e919f is joining an existing cluster
I20260808 15:55:38.143854 2911 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.254.60:43385
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.1@KRBTEST.COM
I20260808 15:55:38.162757 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.134991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.27.254.1:40339)
0808 15:55:38.141384 (+ 6393us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.141395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.141447 (+ 52us) server_negotiation.cc:407] Connection header received
0808 15:55:38.141520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.141529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.141792 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.141959 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.143497 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.145084 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.147151 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.147992 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.148141 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.154301 (+ 6160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.154316 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.154333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.154382 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.158439 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.159354 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.159363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.159366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.159536 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.160023 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.160028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.160031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.160469 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.160779 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.162034 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.162323 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6196,"thread_start_us":6030,"threads_started":1}
I20260808 15:55:38.164913 2531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:38.173183 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.151551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.27.254.61:35737)
0808 15:55:38.151735 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.151741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.153063 (+ 1322us) server_negotiation.cc:407] Connection header received
0808 15:55:38.153511 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.153520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.153639 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.153729 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.155587 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.157225 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.159891 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.160559 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.160704 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.163621 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.163628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.163632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.163672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.167375 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.170288 (+ 2913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.170294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.170298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.170451 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.170928 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.170933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.170937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.171292 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.171426 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.172306 (+ 880us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.172629 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:55:38.174660 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.152223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41404)
0808 15:55:38.152937 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.152963 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.153277 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.153994 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.154002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.154855 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.155346 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.155376 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.157466 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.157475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.159723 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.159737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.160991 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.160996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.161213 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.161942 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.161971 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.163404 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.169585 (+ 6181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.169592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.169596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.170145 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.170604 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.170609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.170615 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.170800 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.171503 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.171508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.171867 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.173480 (+ 1613us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.173913 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":383,"threads_started":1}
I20260808 15:55:38.182098 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.151531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.27.254.62:46835)
0808 15:55:38.151707 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.151711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.159997 (+ 8286us) server_negotiation.cc:407] Connection header received
0808 15:55:38.161374 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.161381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.161500 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.161589 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.165449 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.166959 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.171925 (+ 4966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.172605 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.172716 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.175332 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.175339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.175344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.175383 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.178842 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.179677 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.179682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.179686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.179880 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.180366 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.180372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.180375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.180727 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.180853 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.181530 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.181768 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:55:38.183055 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.153161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54484)
0808 15:55:38.159892 (+ 6731us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.159915 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.161052 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.161858 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.161864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.162605 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.164914 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.164927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.167260 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.167268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.171742 (+ 4474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.171755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.173010 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.173016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.173216 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.173870 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.173891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.175089 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.179023 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.179028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.179032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.179530 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.180031 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.180036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.180039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.180237 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.180952 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.180956 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.181192 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.182297 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.182629 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6563,"thread_start_us":6846,"threads_started":1}
I20260808 15:55:38.196314 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.160491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.27.254.60:43385)
0808 15:55:38.160689 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.160694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.177055 (+ 16361us) server_negotiation.cc:407] Connection header received
0808 15:55:38.177298 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.177305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.177383 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.177488 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.179517 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.181010 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.186188 (+ 5178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.186927 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.187022 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.189389 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.189395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.189400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.189455 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.193292 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.194115 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.194130 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.194134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.194253 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.194632 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.194636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.194639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.195040 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.195201 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.195725 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.195933 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:55:38.197651 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.160807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.0.0.1:41098)
0808 15:55:38.176950 (+ 16143us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.176972 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.177162 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.177718 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.177725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.178417 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.178663 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.178678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.181301 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.181309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.186034 (+ 4725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.186048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.187295 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.187301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.187528 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.188233 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.188257 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.189206 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.193467 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.193472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.193475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.193998 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.194388 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.194393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.194396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.194528 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.195284 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.195288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.195445 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.196988 (+ 1543us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.197270 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16002,"thread_start_us":18452,"threads_started":1}
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204535, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.62@KRBTEST.COM for oryx/127.27.254.60@KRBTEST.COM
I20260808 15:55:38.213653 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.190346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.27.254.60:43385)
0808 15:55:38.196545 (+ 6199us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.196551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.196568 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:38.196654 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.196660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.196745 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.196843 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.198257 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.199692 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.201395 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.202622 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.202791 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.206592 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.206599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.206604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.206639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.210364 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.211306 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.211312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.211316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.211450 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.211947 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.211952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.211955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.212310 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.212449 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.213006 (+ 557us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.213244 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6095,"thread_start_us":14485,"threads_started":1}
I20260808 15:55:38.213665 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.189987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.0.0.1:41108)
0808 15:55:38.190167 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.190187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.190293 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.196911 (+ 6618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.196919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.197544 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.197792 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.197805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.199841 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.199847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.201242 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.201257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.202804 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.202812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.202996 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.203662 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.203685 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.206374 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.210521 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.210528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.210533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.211157 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.211589 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.211596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.211601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.211820 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.212447 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.212454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.212685 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.212887 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.213251 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 15:55:38.215797 2345 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.254.60:43385: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
member_type: NON_VOTER
last_known_addr {
host: "127.27.254.60"
port: 43385
}
attrs {
promote: true
}
}
dest_uuid: "993b6c092dd04040b740a1bfc66d0225"
cas_config_opid_index: 5
I20260808 15:55:38.216809 2345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } }
I20260808 15:55:38.218667 2907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.222182 2957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 LEARNER]: Refusing update from remote peer 993b6c092dd04040b740a1bfc66d0225: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 15:55:38.223968 2907 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.254.60:43385
I20260808 15:55:38.224541 2911 master.cc:537] Master@127.27.254.60:43385 shutting down...
I20260808 15:55:38.233886 3096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260808 15:55:38.238302 2331 consensus_peers.cc:487] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 -> Peer c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60:43385): Couldn't send request to peer c3259a5e0d32465faba0cfb4fe1e919f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 15:55:38.242995 3096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.243330 3096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:38.249939 2957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.254586 2985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: NON_VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: true } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.254880 2985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:55:38.255556 3095 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: attempting to promote NON_VOTER a66c7b522c5e45a096157f6b545f2143 to VOTER
I20260808 15:55:38.256433 3095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } }
I20260808 15:55:38.258731 2957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 LEARNER]: Refusing update from remote peer 993b6c092dd04040b740a1bfc66d0225: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 15:55:38.259291 3096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 15:55:38.259588 2331 consensus_peers.cc:487] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 -> Peer c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60:43385): Couldn't send request to peer c3259a5e0d32465faba0cfb4fe1e919f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 15:55:38.261639 3095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a66c7b522c5e45a096157f6b545f2143 (127.27.254.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.262471 2957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a66c7b522c5e45a096157f6b545f2143 (127.27.254.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.264109 2911 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:38.264523 2911 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:38.264739 2911 tablet_replica.cc:330] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: stopping tablet replica
I20260808 15:55:38.266120 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.266350 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:38.266657 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.266674 2985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.266878 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:38.266893 2985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:55:38.282186 2911 master.cc:552] Master@127.27.254.60:43385 shutdown complete.
I20260808 15:55:38.282285 2911 master_runner.cc:305] Clearing existing system tablet
I20260808 15:55:38.282653 2911 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:38.282764 2911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:38.285784 2911 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260808 15:55:38.293344 3101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:38.293704 2911 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:38.293947 2911 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260808 15:55:38.294044 2911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "4e804b57da788421a109c1d4c7d40b15"
server_key_iv: "785c8bb1ef86660e9d6125cf03f80e83"
server_key_version: "encryptionkey@0"
I20260808 15:55:38.300052 2911 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:38.307183 2911 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 15:55:38.307349 2911 log.cc:1196] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 15:55:38.307811 2911 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Deleting consensus metadata
I20260808 15:55:38.308043 2911 master_runner.cc:315] Copying system tablet from 127.27.254.62:46835
I20260808 15:55:38.311201 2911 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.254.62:46835
I20260808 15:55:38.327171 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.311995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.27.254.62:46835)
0808 15:55:38.312237 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.312242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.312760 (+ 518us) server_negotiation.cc:407] Connection header received
0808 15:55:38.312920 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.312927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.313012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.313127 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.314153 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.315399 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.317363 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.318023 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.318176 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.320657 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.320664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.320669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.320709 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.324190 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.324988 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.324993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.324997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.325150 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.325611 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.325616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.325619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.325924 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.326072 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.326509 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.326795 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 15:55:38.327587 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.311996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54488)
0808 15:55:38.312626 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.312650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.312785 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.313172 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.313177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.313709 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.313979 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.313992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.315607 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.315612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.317198 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.317214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.318222 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.318228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.318436 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.319301 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.319325 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.320432 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.324330 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.324341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.324345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.324862 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.325283 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.325292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.325296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.325492 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.326049 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.326058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.326244 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.326407 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.326812 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":357,"threads_started":1}
I20260808 15:55:38.328343 2366 tablet_copy_service.cc:140] P 993b6c092dd04040b740a1bfc66d0225: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c3259a5e0d32465faba0cfb4fe1e919f ({username='oryx', principal='oryx/127.27.254.60@KRBTEST.COM'} at 127.0.0.1:54488)
I20260808 15:55:38.328526 2366 tablet_copy_service.cc:161] P 993b6c092dd04040b740a1bfc66d0225: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c3259a5e0d32465faba0cfb4fe1e919f at {username='oryx', principal='oryx/127.27.254.60@KRBTEST.COM'} at 127.0.0.1:54488: session id = c3259a5e0d32465faba0cfb4fe1e919f-00000000000000000000000000000000
I20260808 15:55:38.332304 2366 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 15:55:38.334221 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 15:55:38.341105 2911 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: tablet copy: Starting download of 0 data blocks...
I20260808 15:55:38.341398 2911 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: tablet copy: Starting download of 1 WAL segments...
I20260808 15:55:38.342984 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2408
I20260808 15:55:38.343704 2911 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 15:55:38.348133 2366 tablet_copy_service.cc:342] P 993b6c092dd04040b740a1bfc66d0225: Request end of tablet copy session c3259a5e0d32465faba0cfb4fe1e919f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.254.60@KRBTEST.COM'} at 127.0.0.1:54488
I20260808 15:55:38.348286 2366 tablet_copy_service.cc:433] P 993b6c092dd04040b740a1bfc66d0225: ending tablet copy session c3259a5e0d32465faba0cfb4fe1e919f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c3259a5e0d32465faba0cfb4fe1e919f
I20260808 15:55:38.359751 2911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:38.362797 3108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:38.365068 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:40339
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:46835
--webserver_port=34807
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:38.371804 3111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:38.372469 2911 server_base.cc:1034] running on GCE node
W20260808 15:55:38.374089 3109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:38.374571 2911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:38.375295 2911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:38.376546 2911 hybrid_clock.cc:648] HybridClock initialized: now 1786204538376512 us; error 29 us; skew 500 ppm
I20260808 15:55:38.376778 2911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204538, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:38.379051 2911 init.cc:377] Logged in from keytab as oryx/127.27.254.60@KRBTEST.COM (short username oryx)
I20260808 15:55:38.382505 2911 webserver.cc:460] Webserver started at http://127.27.254.60:40313/ using document root <none> and password file <none>
I20260808 15:55:38.382876 2911 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:38.382985 2911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:38.386554 2911 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 15:55:38.388717 3119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:38.389173 2911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260808 15:55:38.389315 2911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
format_stamp: "Formatted at 2026-08-08 15:55:37 on dist-test-slave-srgv"
server_key: "4e804b57da788421a109c1d4c7d40b15"
server_key_iv: "785c8bb1ef86660e9d6125cf03f80e83"
server_key_version: "encryptionkey@0"
I20260808 15:55:38.389541 2911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:38.405109 2911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:38.409082 2911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:38.430402 2911 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.60:43385
I20260808 15:55:38.430549 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.60:43385 every 8 connection(s)
I20260808 15:55:38.431435 2911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
I20260808 15:55:38.436641 3172 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:55:38.440197 3172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Bootstrap starting.
I20260808 15:55:38.458797 3172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:55:38.459393 3172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: Bootstrap complete.
I20260808 15:55:38.460564 3172 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } }
I20260808 15:55:38.460927 3172 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3259a5e0d32465faba0cfb4fe1e919f, State: Initialized, Role: LEARNER
I20260808 15:55:38.461226 3172 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } }
I20260808 15:55:38.462534 3172 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:38.462241 3175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.462947 3175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: This master's current role is: LEARNER
I20260808 15:55:38.478565 3186 catalog_manager.cc:1259] Loaded cluster ID: ed6059a0fcae44d1bae514f36c3a0794
I20260808 15:55:38.478681 3186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: loading cluster ID for follower catalog manager: success
I20260808 15:55:38.482069 3186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: acquiring CA information for follower catalog manager: success
I20260808 15:55:38.484344 3186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:55:38.726008 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.718848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.0.0.1:41116)
0808 15:55:38.719392 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.719420 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.719644 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.720458 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.720462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.720486 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:38.720647 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.720654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.722462 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.722467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.723660 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.723672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.725313 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.725318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.725549 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.725576 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.725657 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.725760 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260808 15:55:38.726217 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.718974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.27.254.60:43385)
0808 15:55:38.720083 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.720090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.720113 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:55:38.720201 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.720207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.720295 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.720430 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:38.720786 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.722306 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.723815 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.725127 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.725280 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.725530 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.725702 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.725846 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":794,"threads_started":1}
I20260808 15:55:38.727404 3147 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 15:55:38.728161 3175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993b6c092dd04040b740a1bfc66d0225. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:38.728454 3175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: This master's current role is: LEARNER
W20260808 15:55:38.742921 3113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:38.743214 3113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:38.743376 3113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:38.750988 3113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:38.751120 3113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:38.751354 3113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:38.761498 3113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:40339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=34807
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:38.762835 3113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:38.764132 3113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:38.777918 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:38.778039 3196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:38.778013 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:38.778792 3113 server_base.cc:1034] running on GCE node
I20260808 15:55:38.779382 3113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:38.780272 3113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:38.781534 3113 hybrid_clock.cc:648] HybridClock initialized: now 1786204538781484 us; error 28 us; skew 500 ppm
I20260808 15:55:38.781888 3113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204538, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:38.785256 3113 init.cc:377] Logged in from keytab as oryx/127.27.254.1@KRBTEST.COM (short username oryx)
I20260808 15:55:38.786724 3113 webserver.cc:460] Webserver started at http://127.27.254.1:34807/ using document root <none> and password file <none>
I20260808 15:55:38.787423 3113 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:38.787586 3113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:38.793138 3113 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:55:38.795855 3203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:38.797245 3113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:38.797497 3113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "d72fbc8141f448ee82771422f4411e37"
format_stamp: "Formatted at 2026-08-08 15:55:35 on dist-test-slave-srgv"
server_key: "9ed2a22cb38086765cb4dd73af6a3a5b"
server_key_iv: "910b68c734d955fb531d226ccd4e6606"
server_key_version: "encryptionkey@0"
I20260808 15:55:38.798480 3113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:38.815353 3113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:38.819681 3113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:38.819999 3113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:38.820966 3113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:38.822458 3113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:38.822578 3113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:38.822710 3113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:38.822803 3113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:38.859704 3113 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:40339
I20260808 15:55:38.859755 3316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:40339 every 8 connection(s)
I20260808 15:55:38.861146 3113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:38.863662 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 3113
I20260808 15:55:38.864043 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2542
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204538, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.60@KRBTEST.COM
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204538, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
I20260808 15:55:38.894407 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.865469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:36079 (local address 127.27.254.60:43385)
0808 15:55:38.865734 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.865743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.874730 (+ 8987us) server_negotiation.cc:407] Connection header received
0808 15:55:38.877191 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.877198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.877298 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.877355 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.879188 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.879924 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.881551 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.881980 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.882121 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.886602 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.886613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.886617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.886655 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.890175 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.891267 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.891274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.891278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.891522 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.892107 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.892113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.892116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.892479 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.892645 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.893739 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.893995 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204538, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
Aug 08 15:55:38 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.2@KRBTEST.COM
I20260808 15:55:38.896629 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.864719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.27.254.1:36079)
0808 15:55:38.874579 (+ 9860us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.874615 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.876865 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.877858 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.877867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.878420 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.879009 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.879035 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.880096 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.880106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.881334 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.881345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.882215 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.882222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.882443 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.883161 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.883188 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.886344 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.890407 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.890418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.890440 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.891124 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.891692 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.891699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.891706 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.891963 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.892752 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.892760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.893240 (+ 480us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.894617 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.895175 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9590,"thread_start_us":10506,"threads_started":1}
I20260808 15:55:38.898579 3317 heartbeater.cc:347] Connected to a master server at 127.27.254.60:43385
I20260808 15:55:38.898965 3317 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:38.899914 3317 heartbeater.cc:511] Master 127.27.254.60:43385 requested a full tablet report, sending...
I20260808 15:55:38.902498 3136 ts_manager.cc:194] Registered new tserver with Master: d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339)
I20260808 15:55:38.905062 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.865579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:53461 (local address 127.27.254.61:35737)
0808 15:55:38.885260 (+ 19681us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.885268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.885293 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:55:38.885350 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.885356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.885465 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.885575 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.886937 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.888128 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.890037 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.890703 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.890824 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.896329 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.896335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.896340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.896382 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.901915 (+ 5533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.902918 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.902923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.902926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.903084 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.903527 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.903532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.903535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.903881 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.904054 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.904471 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.904684 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19551,"thread_start_us":19425,"threads_started":1}
I20260808 15:55:38.905097 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.864846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.27.254.1:53461)
0808 15:55:38.875675 (+ 10829us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.875697 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.877194 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.885821 (+ 8627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.885829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.886437 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.886754 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.886764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.888298 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.888306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.889844 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.889858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.891092 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.891098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.891273 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.892032 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.892053 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.896093 (+ 4040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.902099 (+ 6006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.902107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.902110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.902768 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.903228 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.903232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.903235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.903384 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.904015 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.904022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.904195 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.904371 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.904647 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10697,"thread_start_us":9403,"threads_started":1}
I20260808 15:55:38.906057 3319 heartbeater.cc:347] Connected to a master server at 127.27.254.61:35737
I20260808 15:55:38.906194 3319 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:38.906225 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.866053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.27.254.2:35825)
0808 15:55:38.882292 (+ 16239us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.882304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.882341 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:55:38.882422 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.882434 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.882771 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.885224 (+ 2453us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.885533 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.887257 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.889397 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.890354 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.891741 (+ 1387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.897064 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.897078 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.897094 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.897137 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.902048 (+ 4911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.902951 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.902958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.902963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.903102 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.903542 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.903546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.903549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.903935 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.905363 (+ 1428us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.905553 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.905811 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16045,"thread_start_us":15888,"threads_started":1}
I20260808 15:55:38.906638 3319 heartbeater.cc:511] Master 127.27.254.61:35737 requested a full tablet report, sending...
I20260808 15:55:38.906787 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.875985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:43957 (local address 127.27.254.62:46835)
0808 15:55:38.886359 (+ 10374us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:38.886364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:38.886386 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:55:38.886447 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:38.886452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:38.886543 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:38.887322 (+ 779us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:38.888724 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.889932 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.891927 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.892617 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.894225 (+ 1608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.897770 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:38.897777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:38.897781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:38.897820 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:38.903104 (+ 5284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.903970 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.903978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.903981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.904122 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:38.904593 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:38.904599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:38.904602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:38.904952 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:38.906038 (+ 1086us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:38.906167 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 15:55:38.906414 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10249,"thread_start_us":10105,"threads_started":1}
I20260808 15:55:38.907209 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:38.875828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:43957)
0808 15:55:38.876377 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:38.876395 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:38.876865 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:38.886751 (+ 9886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:38.886756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:38.887649 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:38.887957 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.887968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.890095 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.890100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.891751 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:38.891765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:38.892851 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:38.892858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:38.893064 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:38.893673 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:38.893701 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:38.897551 (+ 3850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:38.903248 (+ 5697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.903253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.903256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.903813 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.904264 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:38.904269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:38.904272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:38.904461 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:38.905646 (+ 1185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:38.905651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:38.905804 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 15:55:38.905940 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 15:55:38.906548 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":1314,"threads_started":1}
I20260808 15:55:38.907485 2665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:38.908437 3318 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46835
I20260808 15:55:38.908391 2947 ts_manager.cc:194] Registered new tserver with Master: d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339)
I20260808 15:55:38.908560 3318 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:38.909000 3318 heartbeater.cc:511] Master 127.27.254.62:46835 requested a full tablet report, sending...
I20260808 15:55:38.910422 2345 ts_manager.cc:194] Re-registered known tserver with Master: d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339)
I20260808 15:55:38.911909 2345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:43957
I20260808 15:55:38.992177 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2542
I20260808 15:55:39.009330 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:35825
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:46835
--webserver_port=43839
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:55:39.310937 3095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: NON_VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: true } } }
I20260808 15:55:39.311383 3095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:39.313118 3095 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: attempting to promote NON_VOTER c3259a5e0d32465faba0cfb4fe1e919f to VOTER
I20260808 15:55:39.313774 3095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } }
I20260808 15:55:39.315896 2957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Refusing update from remote peer 993b6c092dd04040b740a1bfc66d0225: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 15:55:39.316007 3147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 LEARNER]: Refusing update from remote peer 993b6c092dd04040b740a1bfc66d0225: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 15:55:39.316533 3095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 15:55:39.316907 3331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 15:55:39.319943 3331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.320819 2957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.323911 3330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.324258 3330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:39.324563 3330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.324772 3330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:39.330899 3336 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.331269 3336 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:55:39.331516 3095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.331808 3095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:39.332674 3147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.337452 3341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "993b6c092dd04040b740a1bfc66d0225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:39.337800 3341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: This master's current role is: FOLLOWER
W20260808 15:55:39.342547 3329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:39.342864 3329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:39.343026 3329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:39.350441 3329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:39.350587 3329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:39.350755 3329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:39.360512 3329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:35825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=43839
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:39.361857 3329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:39.363173 3329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:39.376287 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:39.376364 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:39.376335 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:39.377002 3329 server_base.cc:1034] running on GCE node
I20260808 15:55:39.377658 3329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:39.378547 3329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:39.379773 3329 hybrid_clock.cc:648] HybridClock initialized: now 1786204539379735 us; error 48 us; skew 500 ppm
I20260808 15:55:39.380081 3329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:39 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:39.383185 3329 init.cc:377] Logged in from keytab as oryx/127.27.254.2@KRBTEST.COM (short username oryx)
I20260808 15:55:39.384626 3329 webserver.cc:460] Webserver started at http://127.27.254.2:43839/ using document root <none> and password file <none>
I20260808 15:55:39.385360 3329 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:39.385545 3329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:39.391305 3329 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 15:55:39.394119 3353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:39.395574 3329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:39.395749 3329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "620e184eeaf44ad19d7c4dd6140af409"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "09fcfdb607f345da6e115eddae5aafb6"
server_key_iv: "5b946405a4a75b4b071983e2d491d465"
server_key_version: "encryptionkey@0"
I20260808 15:55:39.396704 3329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:39.413189 3329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:39.417733 3329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:39.418028 3329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:39.419090 3329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:39.420478 3329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:39.420620 3329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:39.420742 3329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:39.420835 3329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:39.458539 3329 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:35825
I20260808 15:55:39.458593 3466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:35825 every 8 connection(s)
I20260808 15:55:39.460213 3329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:39.469612 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 3329
I20260808 15:55:39.469910 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2675
Aug 08 15:55:39 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
Aug 08 15:55:39 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.60@KRBTEST.COM
I20260808 15:55:39.490576 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.464300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:45485 (local address 127.27.254.61:35737)
0808 15:55:39.469065 (+ 4765us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:39.469070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:39.469085 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:55:39.469180 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:39.469185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:39.469279 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:39.471700 (+ 2421us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:39.471828 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.472888 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.475442 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.476050 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.476167 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.481164 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:39.481171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:39.481175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:39.481224 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:39.485883 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.486991 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.487000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.487004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.487158 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.487723 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.487728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.487731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.488106 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:39.488269 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:39.489910 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0808 15:55:39.490216 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4654,"thread_start_us":4561,"threads_started":1}
I20260808 15:55:39.491211 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.463116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44827 (local address 127.27.254.60:43385)
0808 15:55:39.463697 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:39.463702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:39.472074 (+ 8372us) server_negotiation.cc:407] Connection header received
0808 15:55:39.472240 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:39.472245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:39.472332 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:39.472436 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:39.473751 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.474913 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.476805 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.477592 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.477699 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.482313 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:39.482322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:39.482327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:39.482366 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:39.485929 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.488870 (+ 2941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.488876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.488879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.489034 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.489595 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.489602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.489606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.489944 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:39.490112 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:39.490640 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 15:55:39.490863 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":320,"threads_started":1}
I20260808 15:55:39.492161 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.463774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.27.254.2:45485)
0808 15:55:39.464805 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:39.464872 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:39.468327 (+ 3455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:39.469844 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:39.469858 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:39.470744 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:39.471511 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:39.471545 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.473078 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.473084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:39.475235 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:39.475251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.476446 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.476459 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:39.476739 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.477559 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:39.477589 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:39.480918 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:39.486043 (+ 5125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:39.486060 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:39.486078 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:39.486851 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:39.487300 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:39.487305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:39.487309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:39.487572 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:39.488347 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:39.488357 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:39.488847 (+ 490us) client_negotiation.cc:769] Sending connection context
0808 15:55:39.489185 (+ 338us) client_negotiation.cc:240] Negotiation successful
0808 15:55:39.490222 (+ 1037us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":3,"spinlock_wait_cycles":146048,"thread_start_us":473,"threads_started":1}
I20260808 15:55:39.492673 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.464328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.27.254.2:44827)
0808 15:55:39.471929 (+ 7601us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:39.471954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:39.472116 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:39.472609 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:39.472613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:39.473189 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:39.473568 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:39.473579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.475102 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.475107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:39.476652 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:39.476665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.478022 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.478031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:39.478224 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.478838 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:39.478859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:39.482066 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:39.486075 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:39.486078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:39.486081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:39.488726 (+ 2645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:39.489193 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:39.489198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:39.489201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:39.489433 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:39.490196 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:39.490200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:39.490344 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 15:55:39.491386 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0808 15:55:39.491675 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1329,"mutex_wait_us":95,"thread_start_us":1042,"threads_started":1}
I20260808 15:55:39.493821 3469 heartbeater.cc:347] Connected to a master server at 127.27.254.61:35737
I20260808 15:55:39.494150 3469 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:39.495090 3469 heartbeater.cc:511] Master 127.27.254.61:35737 requested a full tablet report, sending...
I20260808 15:55:39.495288 3467 heartbeater.cc:347] Connected to a master server at 127.27.254.60:43385
I20260808 15:55:39.495430 3467 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:39.495877 3467 heartbeater.cc:511] Master 127.27.254.60:43385 requested a full tablet report, sending...
I20260808 15:55:39.496829 3136 ts_manager.cc:194] Registered new tserver with Master: 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825)
I20260808 15:55:39.496917 2947 ts_manager.cc:194] Registered new tserver with Master: 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825)
Aug 08 15:55:39 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.3@KRBTEST.COM
Aug 08 15:55:39 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:39.505816 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.471445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.27.254.3:43015)
0808 15:55:39.486156 (+ 14711us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:39.486169 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:39.486211 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:55:39.486299 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:39.486311 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:39.486643 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:39.486868 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:39.488999 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.490550 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.492454 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.493316 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.494252 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.499634 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:39.499645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:39.499656 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:39.499692 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:39.502936 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.503737 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.503743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.503745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.503865 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.504269 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.504274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.504278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.504663 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:39.504964 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:39.505211 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 15:55:39.505398 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14446,"thread_start_us":15655,"threads_started":1}
I20260808 15:55:39.506423 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.465896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:48449)
0808 15:55:39.487865 (+ 21969us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:39.487890 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:39.488027 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:39.489043 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:39.489048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:39.489833 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:39.491904 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:39.491920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.493433 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.493439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:39.495013 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:39.495027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.496526 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.496533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:39.496710 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.498319 (+ 1609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:39.498335 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:39.500315 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:39.503963 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:39.503971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:39.503974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:39.504393 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:39.504782 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:39.504789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:39.504792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:39.504964 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:39.505645 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:39.505649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:39.505803 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 15:55:39.505945 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 15:55:39.506173 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21805,"thread_start_us":22495,"threads_started":1}
I20260808 15:55:39.507160 2800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:39.507459 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:39.466023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:48449 (local address 127.27.254.62:46835)
0808 15:55:39.488531 (+ 22508us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:39.488537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:39.488558 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:55:39.488624 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:39.488631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:39.488712 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:39.488819 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:39.492089 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.493243 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.495254 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:39.496197 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:39.496337 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:39.500517 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:39.500525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:39.500530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:39.500605 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:39.503845 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.504531 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.504534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.504542 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.504669 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:39.505105 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:39.505110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:39.505113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:39.505488 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:39.505644 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:39.506948 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0808 15:55:39.507101 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22395,"thread_start_us":22239,"threads_started":1}
I20260808 15:55:39.508463 3468 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46835
I20260808 15:55:39.508605 3468 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:39.509163 3468 heartbeater.cc:511] Master 127.27.254.62:46835 requested a full tablet report, sending...
I20260808 15:55:39.510319 2345 ts_manager.cc:194] Re-registered known tserver with Master: 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825)
I20260808 15:55:39.511140 2345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:48449
I20260808 15:55:39.591482 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2675
I20260808 15:55:39.610117 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:43015
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:46835
--webserver_port=36987
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:55:39.914588 3318 heartbeater.cc:503] Master 127.27.254.62:46835 was elected leader, sending a full tablet report...
W20260808 15:55:39.943002 3480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:39.943316 3480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:39.943470 3480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:39.954500 3480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:39.954604 3480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:39.954712 3480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:39.964443 3480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:43015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=36987
--tserver_master_addrs=127.27.254.62:46835,127.27.254.61:35737,127.27.254.60:43385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:39.965847 3480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:39.967146 3480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:39.980104 3486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:39.980063 3488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:39.980147 3485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:39.982789 3480 server_base.cc:1034] running on GCE node
I20260808 15:55:39.983433 3480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:39.984335 3480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:39.985513 3480 hybrid_clock.cc:648] HybridClock initialized: now 1786204539985464 us; error 33 us; skew 500 ppm
I20260808 15:55:39.985826 3480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:39 dist-test-slave-srgv krb5kdc[2301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:39.988732 3480 init.cc:377] Logged in from keytab as oryx/127.27.254.3@KRBTEST.COM (short username oryx)
I20260808 15:55:39.990346 3480 webserver.cc:460] Webserver started at http://127.27.254.3:36987/ using document root <none> and password file <none>
I20260808 15:55:39.991086 3480 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:39.991242 3480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:39.996409 3480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:55:39.999006 3495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:40.000330 3480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:55:40.000494 3480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "e9f32ff8506f4806ab40be69e08ec0d0"
format_stamp: "Formatted at 2026-08-08 15:55:36 on dist-test-slave-srgv"
server_key: "47ebf0bd2a6e80c847f3090d25395488"
server_key_iv: "fdfddfe79907cf1eb01793b496fd950c"
server_key_version: "encryptionkey@0"
I20260808 15:55:40.001566 3480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:40.015137 3480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:40.018988 3480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:40.019269 3480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:40.020190 3480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:40.021616 3480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:40.021735 3480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:40.021852 3480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:40.021944 3480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:40.054637 3480 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:43015
I20260808 15:55:40.054724 3609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:43015 every 8 connection(s)
I20260808 15:55:40.056077 3480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:40.060501 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 3480
Aug 08 15:55:40 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.61@KRBTEST.COM
I20260808 15:55:40.086059 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.059658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.27.254.3:41649)
0808 15:55:40.060830 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.060883 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.063664 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.064557 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.064567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.065435 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:40.066496 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.066530 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.068565 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.068575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.071376 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.071396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.073080 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.073090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.073502 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.074415 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:40.074444 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:40.076806 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:40.080507 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:40.080516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:40.080533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:40.081178 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:40.081830 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:40.081837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:40.081844 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:40.082137 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:40.083012 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:40.083023 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:40.083522 (+ 499us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.083857 (+ 335us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.084437 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":405,"threads_started":1}
I20260808 15:55:40.087407 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.059011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:41649 (local address 127.27.254.61:35737)
0808 15:55:40.059575 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.059579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.060985 (+ 1406us) server_negotiation.cc:407] Connection header received
0808 15:55:40.063983 (+ 2998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.063992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.064104 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.064173 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.066752 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.068289 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.071591 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.072522 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.072645 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.077035 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.077040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.077043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.077080 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.080365 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.081371 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.081380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.081386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.081653 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.082299 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.082307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.082313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.082760 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.082934 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.086661 (+ 3727us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.086920 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":963,"threads_started":1}
I20260808 15:55:40.088692 3612 heartbeater.cc:347] Connected to a master server at 127.27.254.61:35737
I20260808 15:55:40.089027 3612 heartbeater.cc:464] Registering TS with master...
Aug 08 15:55:40 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.60@KRBTEST.COM
I20260808 15:55:40.090006 3612 heartbeater.cc:511] Master 127.27.254.61:35737 requested a full tablet report, sending...
I20260808 15:55:40.091969 2947 ts_manager.cc:194] Registered new tserver with Master: e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015)
I20260808 15:55:40.098193 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.063014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.27.254.3:56787)
0808 15:55:40.077843 (+ 14829us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.077862 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.077973 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.078774 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.078781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.079360 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:40.079718 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.079732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.081347 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.081355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.083485 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.083504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.085519 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.085529 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.085790 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.086877 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:40.086908 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:40.090221 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:40.095322 (+ 5101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:40.095326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:40.095330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:40.095792 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:40.096245 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:40.096252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:40.096261 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:40.096405 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:40.097118 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:40.097123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:40.097272 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.097428 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.097757 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14702,"thread_start_us":14573,"threads_started":1}
I20260808 15:55:40.101318 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.062297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:56787 (local address 127.27.254.60:43385)
0808 15:55:40.078427 (+ 16130us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.078433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.078453 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:55:40.078512 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.078517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.078601 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.078713 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.079898 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.081124 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.083697 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.084581 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.084738 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.090402 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.090408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.090413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.090453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.095160 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.095920 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.095925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.095929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.096101 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.096539 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.096544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.096548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.096901 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.097033 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.100703 (+ 3670us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.100934 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16006,"thread_start_us":15895,"threads_started":1}
I20260808 15:55:40.102281 3610 heartbeater.cc:347] Connected to a master server at 127.27.254.60:43385
I20260808 15:55:40.102397 3610 heartbeater.cc:464] Registering TS with master...
Aug 08 15:55:40 dist-test-slave-srgv krb5kdc[2301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204539, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:40.102888 3610 heartbeater.cc:511] Master 127.27.254.60:43385 requested a full tablet report, sending...
I20260808 15:55:40.105855 3136 ts_manager.cc:194] Registered new tserver with Master: e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015)
I20260808 15:55:40.112450 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.066295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.27.254.61:35737)
0808 15:55:40.087683 (+ 21388us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.087689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.089434 (+ 1745us) server_negotiation.cc:407] Connection header received
0808 15:55:40.090564 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.090571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.090663 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.090733 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.093299 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.094517 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.098728 (+ 4211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.099382 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.099583 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.104679 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.104685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.104689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.104725 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.108841 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.109875 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.109883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.109888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.110071 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.110592 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.110597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.110601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.110973 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.111223 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.111781 (+ 558us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.112038 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21259,"thread_start_us":28444,"threads_started":1}
I20260808 15:55:40.114771 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.059658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:37361)
0808 15:55:40.061971 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.061989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.063672 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.093456 (+ 29784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.093461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.093964 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:40.094154 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.094162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.095946 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.095951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.097517 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.097529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.099684 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.099694 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.099902 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.100488 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:40.100510 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:40.103220 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:40.111792 (+ 8572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:40.111797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:40.111801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:40.112268 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:40.113134 (+ 866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:40.113139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:40.113142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:40.113316 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:40.113925 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:40.113930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:40.114081 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.114225 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.114487 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2135,"mutex_wait_us":49,"spinlock_wait_cycles":16768,"thread_start_us":1822,"threads_started":1}
I20260808 15:55:40.114977 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.058980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:37361 (local address 127.27.254.62:46835)
0808 15:55:40.059467 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.059473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.093070 (+ 33597us) server_negotiation.cc:407] Connection header received
0808 15:55:40.093172 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.093180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.093275 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.093425 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.094312 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.095452 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.098810 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.099408 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.099588 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.103438 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.103447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.103451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.103490 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.111549 (+ 8059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.112874 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.112879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.112883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.113021 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.113465 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.113469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.113472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.113813 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.113960 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.114351 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.114602 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
I20260808 15:55:40.115757 3611 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46835
I20260808 15:55:40.115892 3611 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:40.116369 3611 heartbeater.cc:511] Master 127.27.254.62:46835 requested a full tablet report, sending...
I20260808 15:55:40.116878 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.065978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.27.254.62:46835)
0808 15:55:40.098211 (+ 32233us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.098215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.098232 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:40.098305 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.098312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.098390 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.098491 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.100629 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.101664 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.103949 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.104621 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.105441 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.108551 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.108557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.108562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.108608 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.112749 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.114052 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.114059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.114064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.114189 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.114653 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.114658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.114661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.115362 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.115750 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.116315 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.116493 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32104,"thread_start_us":31979,"threads_started":1}
I20260808 15:55:40.117489 2345 ts_manager.cc:194] Re-registered known tserver with Master: e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015)
I20260808 15:55:40.118615 2345 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:37361
I20260808 15:55:40.119546 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.080893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.27.254.60:43385)
0808 15:55:40.085332 (+ 4439us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.085340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.097641 (+ 12301us) server_negotiation.cc:407] Connection header received
0808 15:55:40.097865 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.097870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.097942 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.098037 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.104146 (+ 6109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.105310 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.106956 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.107511 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.107651 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.111016 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.111024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.111030 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.111067 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.116064 (+ 4997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.117093 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.117099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.117102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.117281 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.117763 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.117768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.117771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.118096 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.118434 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.118893 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.119118 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4291,"thread_start_us":4139,"threads_started":1}
I20260808 15:55:40.121536 2345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:40.128679 2345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:40.176020 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.151606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.27.254.3:43015)
0808 15:55:40.168075 (+ 16469us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.168089 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.168131 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:55:40.168253 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.168262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.168600 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.168864 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:40.169308 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.170526 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.173572 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.174998 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.175110 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.175263 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.175542 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.175691 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16237,"thread_start_us":17554,"threads_started":1}
I20260808 15:55:40.176049 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.151316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:43015 (local address 127.0.0.1:54992)
0808 15:55:40.162689 (+ 11373us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.162714 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.162854 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.168833 (+ 5979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.168836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.168857 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:40.169120 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.169127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.171084 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.171088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.173386 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.173401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.175213 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.175221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.175370 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.175403 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.175503 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.175647 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10862,"thread_start_us":12127,"threads_started":1}
I20260808 15:55:40.178768 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.150773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40339 (local address 127.0.0.1:36292)
0808 15:55:40.157384 (+ 6611us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.157429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.157575 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.171710 (+ 14135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.171716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.171747 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:40.172178 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.172190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.173869 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.173876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.175569 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.175581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.177972 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.177979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.178173 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.178206 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.178329 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.178458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6425,"thread_start_us":6240,"threads_started":1}
I20260808 15:55:40.178840 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.151026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.27.254.1:40339)
0808 15:55:40.171163 (+ 20137us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.171171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.171202 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:55:40.171324 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.171334 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.171535 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.171724 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:40.172358 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.173575 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.175700 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.177777 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.177905 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.178088 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.178388 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.178540 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19991,"thread_start_us":19868,"threads_started":1}
I20260808 15:55:40.180353 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.151622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35825 (local address 127.0.0.1:59600)
0808 15:55:40.163417 (+ 11795us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.163434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.163535 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.170530 (+ 6995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.170535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.170564 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:40.170891 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.170899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.175817 (+ 4918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.175822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.177553 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.177568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.179604 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.179612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.179780 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.179811 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.179903 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.180027 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11682,"thread_start_us":10746,"threads_started":1}
I20260808 15:55:40.180212 3543 tablet_service.cc:1467] Processing CreateTablet for tablet a0dbef4d45734b058048edff181a834f (DEFAULT_TABLE table=test-table [id=18e2c3c990aa4ef5968fc2819df08482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:40.181042 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.151777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.27.254.2:35825)
0808 15:55:40.169776 (+ 17999us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.169788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.169825 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:55:40.169945 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.169962 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.170293 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.170527 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:40.171079 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.173375 (+ 2296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.177715 (+ 4340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.179370 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.179496 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.179708 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.180620 (+ 912us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.180781 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17815,"thread_start_us":17709,"threads_started":1}
I20260808 15:55:40.181751 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0dbef4d45734b058048edff181a834f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:40.182214 3251 tablet_service.cc:1467] Processing CreateTablet for tablet a0dbef4d45734b058048edff181a834f (DEFAULT_TABLE table=test-table [id=18e2c3c990aa4ef5968fc2819df08482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:40.183518 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0dbef4d45734b058048edff181a834f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:40.184386 3401 tablet_service.cc:1467] Processing CreateTablet for tablet a0dbef4d45734b058048edff181a834f (DEFAULT_TABLE table=test-table [id=18e2c3c990aa4ef5968fc2819df08482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:40.185664 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0dbef4d45734b058048edff181a834f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:40.194770 3645 tablet_bootstrap.cc:492] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Bootstrap starting.
I20260808 15:55:40.197086 3646 tablet_bootstrap.cc:492] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Bootstrap starting.
I20260808 15:55:40.198582 3645 tablet_bootstrap.cc:654] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:40.199635 3645 log.cc:824] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:40.200403 3646 tablet_bootstrap.cc:654] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:40.201960 3646 log.cc:824] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:40.204151 3648 tablet_bootstrap.cc:492] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Bootstrap starting.
I20260808 15:55:40.205639 3646 tablet_bootstrap.cc:492] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: No bootstrap required, opened a new log
I20260808 15:55:40.205915 3646 ts_tablet_manager.cc:1397] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260808 15:55:40.207419 3648 tablet_bootstrap.cc:654] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:40.208544 3648 log.cc:824] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:40.212109 3648 tablet_bootstrap.cc:492] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: No bootstrap required, opened a new log
I20260808 15:55:40.212105 3645 tablet_bootstrap.cc:492] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: No bootstrap required, opened a new log
I20260808 15:55:40.212375 3645 ts_tablet_manager.cc:1397] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20260808 15:55:40.212378 3648 ts_tablet_manager.cc:1397] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 15:55:40.215126 3646 raft_consensus.cc:348] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.215473 3646 raft_consensus.cc:374] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:40.215612 3646 raft_consensus.cc:729] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72fbc8141f448ee82771422f4411e37, State: Initialized, Role: FOLLOWER
I20260808 15:55:40.216207 3646 consensus_queue.cc:260] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.221594 3646 ts_tablet_manager.cc:1428] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
I20260808 15:55:40.223027 3645 raft_consensus.cc:348] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.223027 3648 raft_consensus.cc:348] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.223490 3648 raft_consensus.cc:374] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:40.223490 3645 raft_consensus.cc:374] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:40.223595 3648 raft_consensus.cc:729] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620e184eeaf44ad19d7c4dd6140af409, State: Initialized, Role: FOLLOWER
I20260808 15:55:40.223598 3645 raft_consensus.cc:729] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f32ff8506f4806ab40be69e08ec0d0, State: Initialized, Role: FOLLOWER
I20260808 15:55:40.224232 3648 consensus_queue.cc:260] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.224251 3645 consensus_queue.cc:260] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.226634 3645 ts_tablet_manager.cc:1428] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Time spent starting tablet: real 0.014s user 0.009s sys 0.007s
I20260808 15:55:40.226639 3648 ts_tablet_manager.cc:1428] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260808 15:55:40.226917 3468 heartbeater.cc:503] Master 127.27.254.62:46835 was elected leader, sending a full tablet report...
I20260808 15:55:40.226950 3611 heartbeater.cc:503] Master 127.27.254.62:46835 was elected leader, sending a full tablet report...
W20260808 15:55:40.308986 3613 tablet.cc:2369] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:40.328799 3652 raft_consensus.cc:482] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:40.329095 3652 raft_consensus.cc:504] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.330801 3652 leader_election.cc:288] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339), e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015)
I20260808 15:55:40.337888 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.331342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:33693 (local address 127.27.254.3:43015)
0808 15:55:40.331521 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.331529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.332224 (+ 695us) server_negotiation.cc:407] Connection header received
0808 15:55:40.332461 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.332465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.332524 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.332635 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:40.333310 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.334178 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.336144 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.336883 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.337058 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.337136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.337513 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.337641 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:55:40.337914 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.331288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:57421 (local address 127.27.254.1:40339)
0808 15:55:40.331503 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.331509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.332261 (+ 752us) server_negotiation.cc:407] Connection header received
0808 15:55:40.332473 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.332485 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.332556 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.332654 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:40.333268 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.334150 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.336146 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.336916 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.337079 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.337162 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.337543 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.337661 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:55:40.338064 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.331150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:43015 (local address 127.27.254.2:33693)
0808 15:55:40.332088 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.332113 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.332291 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.332641 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.332646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.332668 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:40.333161 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.333175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.334299 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.334306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.335977 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.335988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.337096 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.337102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.337274 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.337346 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.337487 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.337610 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":45,"thread_start_us":570,"threads_started":1}
I20260808 15:55:40.338303 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.331171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40339 (local address 127.27.254.2:57421)
0808 15:55:40.332148 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:40.332164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:40.332321 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:40.332667 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:40.332671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:40.332694 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:40.333094 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.333103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.334285 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.334290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.335983 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:40.335994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.337152 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.337158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:40.337344 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.337365 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 15:55:40.337502 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 15:55:40.337618 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":7,"thread_start_us":554,"threads_started":1}
I20260808 15:55:40.339198 3563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0dbef4d45734b058048edff181a834f" candidate_uuid: "620e184eeaf44ad19d7c4dd6140af409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" is_pre_election: true
I20260808 15:55:40.339414 3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0dbef4d45734b058048edff181a834f" candidate_uuid: "620e184eeaf44ad19d7c4dd6140af409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72fbc8141f448ee82771422f4411e37" is_pre_election: true
I20260808 15:55:40.339743 3563 raft_consensus.cc:2454] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620e184eeaf44ad19d7c4dd6140af409 in term 0.
I20260808 15:55:40.339920 3271 raft_consensus.cc:2454] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 620e184eeaf44ad19d7c4dd6140af409 in term 0.
I20260808 15:55:40.340547 3357 leader_election.cc:302] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 620e184eeaf44ad19d7c4dd6140af409, d72fbc8141f448ee82771422f4411e37, e9f32ff8506f4806ab40be69e08ec0d0; no voters:
I20260808 15:55:40.340999 3652 raft_consensus.cc:2781] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:40.341143 3652 raft_consensus.cc:482] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:40.341266 3652 raft_consensus.cc:3037] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:40.343442 3652 raft_consensus.cc:504] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.343971 3652 leader_election.cc:288] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 election: Requested vote from peers d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339), e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015)
I20260808 15:55:40.344439 3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0dbef4d45734b058048edff181a834f" candidate_uuid: "620e184eeaf44ad19d7c4dd6140af409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72fbc8141f448ee82771422f4411e37"
I20260808 15:55:40.344485 3563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0dbef4d45734b058048edff181a834f" candidate_uuid: "620e184eeaf44ad19d7c4dd6140af409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f32ff8506f4806ab40be69e08ec0d0"
I20260808 15:55:40.344647 3271 raft_consensus.cc:3037] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:40.344662 3563 raft_consensus.cc:3037] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:40.347028 3271 raft_consensus.cc:2454] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620e184eeaf44ad19d7c4dd6140af409 in term 1.
I20260808 15:55:40.347069 3563 raft_consensus.cc:2454] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 620e184eeaf44ad19d7c4dd6140af409 in term 1.
I20260808 15:55:40.347406 3357 leader_election.cc:302] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 620e184eeaf44ad19d7c4dd6140af409, d72fbc8141f448ee82771422f4411e37; no voters:
I20260808 15:55:40.347713 3652 raft_consensus.cc:2781] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:40.348174 3652 raft_consensus.cc:686] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 1 LEADER]: Becoming Leader. State: Replica: 620e184eeaf44ad19d7c4dd6140af409, State: Running, Role: LEADER
I20260808 15:55:40.348673 3652 consensus_queue.cc:237] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:40.353525 2344 catalog_manager.cc:5495] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 reported cstate change: term changed from 0 to 1, leader changed from <none> to 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "620e184eeaf44ad19d7c4dd6140af409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } health_report { overall_health: HEALTHY } } }
W20260808 15:55:40.371003 3320 tablet.cc:2369] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:40.398448 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.393029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.27.254.2:35825)
0808 15:55:40.393259 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.393265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.393365 (+ 100us) server_negotiation.cc:407] Connection header received
0808 15:55:40.393642 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.393647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.393735 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.393887 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:40.394251 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.394925 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.396246 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.396702 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.396844 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.397317 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:40.397870 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:40.398030 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.398098 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.398188 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 15:55:40.406760 3563 raft_consensus.cc:1261] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Refusing update from remote peer 620e184eeaf44ad19d7c4dd6140af409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:40.406837 3271 raft_consensus.cc:1261] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Refusing update from remote peer 620e184eeaf44ad19d7c4dd6140af409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:40.407747 3652 consensus_queue.cc:1035] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:40.408030 3656 consensus_queue.cc:1035] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:40.421139 3659 mvcc.cc:205] Tried to move back new op lower bound from 7316293797495889920 to 7316293797272276992. Current Snapshot: MvccSnapshot[applied={T|T < 7316293797495889920}]
I20260808 15:55:40.421689 3660 mvcc.cc:205] Tried to move back new op lower bound from 7316293797495889920 to 7316293797272276992. Current Snapshot: MvccSnapshot[applied={T|T < 7316293797495889920}]
I20260808 15:55:40.444293 2344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:40.444561 2344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:40.452509 2344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 993b6c092dd04040b740a1bfc66d0225: Sending DeleteTablet for 3 replicas of tablet a0dbef4d45734b058048edff181a834f
I20260808 15:55:40.453810 3543 tablet_service.cc:1514] Processing DeleteTablet for tablet a0dbef4d45734b058048edff181a834f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:40 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:54992
I20260808 15:55:40.453804 3251 tablet_service.cc:1514] Processing DeleteTablet for tablet a0dbef4d45734b058048edff181a834f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:40 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:36292
I20260808 15:55:40.453944 3401 tablet_service.cc:1514] Processing DeleteTablet for tablet a0dbef4d45734b058048edff181a834f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:40 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:59600
I20260808 15:55:40.475677 3670 tablet_replica.cc:330] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: stopping tablet replica
I20260808 15:55:40.476280 3670 raft_consensus.cc:2229] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:40.476948 3670 raft_consensus.cc:2258] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:40.480656 3670 ts_tablet_manager.cc:1905] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:40.482905 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2314
I20260808 15:55:40.484730 2376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:40.490957 3669 tablet_replica.cc:330] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: stopping tablet replica
I20260808 15:55:40.491401 3669 raft_consensus.cc:2229] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:40.491865 3669 raft_consensus.cc:2258] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:40.492661 3671 tablet_replica.cc:330] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: stopping tablet replica
I20260808 15:55:40.493070 3671 raft_consensus.cc:2229] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:40.493757 3671 raft_consensus.cc:2258] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:40.501997 3671 ts_tablet_manager.cc:1905] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:40.502281 3669 ts_tablet_manager.cc:1905] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:40.503572 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.455714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.27.254.61:35737)
0808 15:55:40.455927 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.455936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.478116 (+ 22180us) server_negotiation.cc:407] Connection header received
0808 15:55:40.478657 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.478666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.478774 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.478901 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.482542 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.483815 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.487609 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.488507 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.488630 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.493612 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.493623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.493631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.493685 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.498537 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.499688 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.499697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.499703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.499949 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.500588 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.500596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.500601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.501208 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.501640 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.502614 (+ 974us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.502946 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:55:40.505033 3670 ts_tablet_manager.cc:1918] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:40.505206 3670 log.cc:1196] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/a0dbef4d45734b058048edff181a834f
I20260808 15:55:40.513111 3670 ts_tablet_manager.cc:1939] T a0dbef4d45734b058048edff181a834f P d72fbc8141f448ee82771422f4411e37: Deleting consensus metadata
I20260808 15:55:40.518417 3669 ts_tablet_manager.cc:1918] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:40.518615 3669 log.cc:1196] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/a0dbef4d45734b058048edff181a834f
I20260808 15:55:40.519168 3669 ts_tablet_manager.cc:1939] T a0dbef4d45734b058048edff181a834f P e9f32ff8506f4806ab40be69e08ec0d0: Deleting consensus metadata
I20260808 15:55:40.525149 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.478326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.27.254.60:43385)
0808 15:55:40.478556 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.478564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.506607 (+ 28043us) server_negotiation.cc:407] Connection header received
0808 15:55:40.507039 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.507048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.507174 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.507264 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.509874 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.511495 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.514642 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.515322 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.515448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.518595 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.518603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.518611 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.518651 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.522048 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.522814 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.522819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.522823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.522990 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.523464 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.523472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.523476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.523888 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.524167 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.524534 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.524770 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:55:40.531387 3671 ts_tablet_manager.cc:1918] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:40.531576 3671 log.cc:1196] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/a0dbef4d45734b058048edff181a834f
I20260808 15:55:40.532196 3671 ts_tablet_manager.cc:1939] T a0dbef4d45734b058048edff181a834f P 620e184eeaf44ad19d7c4dd6140af409: Deleting consensus metadata
I20260808 15:55:40.685587 2333 catalog_manager.cc:4841] TS d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339): tablet a0dbef4d45734b058048edff181a834f (table test-table [id=18e2c3c990aa4ef5968fc2819df08482]) successfully deleted
I20260808 15:55:40.685694 2332 catalog_manager.cc:4841] TS 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825): tablet a0dbef4d45734b058048edff181a834f (table test-table [id=18e2c3c990aa4ef5968fc2819df08482]) successfully deleted
I20260808 15:55:40.685953 2330 catalog_manager.cc:4841] TS e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015): tablet a0dbef4d45734b058048edff181a834f (table test-table [id=18e2c3c990aa4ef5968fc2819df08482]) successfully deleted
I20260808 15:55:40.686580 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2314
I20260808 15:55:40.742960 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.726984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.27.254.60:43385)
0808 15:55:40.727165 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.727171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.727343 (+ 172us) server_negotiation.cc:407] Connection header received
0808 15:55:40.727512 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.727517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.727607 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.727720 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.730090 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.730831 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.732215 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.732905 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.733044 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.736642 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.736650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.736659 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.736692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.739427 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.740066 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.740072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.740076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.740201 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.740685 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.740690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.740692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.741050 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.741299 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.742371 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.742581 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:55:40.743351 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:40.718191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.27.254.61:35737)
0808 15:55:40.718435 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:40.718443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:40.726397 (+ 7954us) server_negotiation.cc:407] Connection header received
0808 15:55:40.726719 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:40.726724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:40.726837 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:40.726933 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:40.729233 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.730069 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.731980 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:40.732476 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:40.732614 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:40.736623 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:40.736630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:40.736635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:40.736687 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:40.740154 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.740893 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.740901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.740908 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.741033 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:40.741386 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:40.741392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:40.741395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:40.741843 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:40.742120 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:40.742849 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 15:55:40.743091 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:55:41.433607 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.432394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:43943)
0808 15:55:41.433166 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.433319 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":553,"threads_started":1}
W20260808 15:55:41.434170 3318 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:46835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.439497 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.438772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:47099)
0808 15:55:41.439198 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.439323 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":235,"threads_started":1}
W20260808 15:55:41.439841 3468 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:46835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.442395 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.441669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:35423)
0808 15:55:41.442079 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.442199 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"threads_started":1}
W20260808 15:55:41.443544 3611 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:46835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.952936 3683 raft_consensus.cc:482] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 993b6c092dd04040b740a1bfc66d0225)
I20260808 15:55:41.953348 3683 raft_consensus.cc:504] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } }
I20260808 15:55:41.953552 3684 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 993b6c092dd04040b740a1bfc66d0225)
I20260808 15:55:41.953819 3684 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } }
I20260808 15:55:41.955089 3684 leader_election.cc:288] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835), a66c7b522c5e45a096157f6b545f2143 (127.27.254.61:35737)
I20260808 15:55:41.958534 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.955272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54950)
0808 15:55:41.957949 (+ 2677us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.958236 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2503,"thread_start_us":2347,"threads_started":1}
W20260808 15:55:41.961615 2932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.962152 3683 leader_election.cc:288] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835), c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60:43385)
I20260808 15:55:41.963841 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.963209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54964)
0808 15:55:41.963453 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.963627 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260808 15:55:41.964002 2932 leader_election.cc:334] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835): Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.964749 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.955268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54934)
0808 15:55:41.964273 (+ 9005us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.964481 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7611,"threads_started":1}
I20260808 15:55:41.969341 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.958374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:43385 (local address 127.0.0.1:41154)
0808 15:55:41.958770 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.958791 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:41.958947 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:41.963036 (+ 4089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:41.963044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:41.963077 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:41.963366 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:41.963379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.965315 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.965320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:41.967187 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:41.967199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.968517 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.968523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:41.968716 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:41.968750 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:55:41.968846 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 15:55:41.968982 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":49,"thread_start_us":2906,"threads_started":1}
I20260808 15:55:41.969370 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.958564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.27.254.60:43385)
0808 15:55:41.961828 (+ 3264us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:41.961834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:41.961852 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:55:41.961928 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:41.961936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:41.962016 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:41.963530 (+ 1514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:41.963832 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.965163 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.967330 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.968359 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.968514 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:41.968769 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:41.968870 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 15:55:41.968996 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":3143,"thread_start_us":2995,"threads_started":1}
I20260808 15:55:41.970819 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a66c7b522c5e45a096157f6b545f2143" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" is_pre_election: true
I20260808 15:55:41.971228 3147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66c7b522c5e45a096157f6b545f2143 in term 1.
I20260808 15:55:41.971752 2932 leader_election.cc:302] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a66c7b522c5e45a096157f6b545f2143, c3259a5e0d32465faba0cfb4fe1e919f; no voters: 993b6c092dd04040b740a1bfc66d0225
I20260808 15:55:41.972096 3683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 15:55:41.972225 3683 raft_consensus.cc:482] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 993b6c092dd04040b740a1bfc66d0225)
I20260808 15:55:41.972324 3683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 1 FOLLOWER]: Advancing to term 2
W20260808 15:55:41.972415 3121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.974421 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.973767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54970)
0808 15:55:41.973954 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.974180 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:55:41.974833 3683 raft_consensus.cc:504] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } }
I20260808 15:55:41.975238 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.955331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.27.254.61:35737)
0808 15:55:41.955989 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:41.955998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:41.965032 (+ 9034us) server_negotiation.cc:407] Connection header received
0808 15:55:41.965197 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:41.965205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:41.965301 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:41.965473 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:41.966058 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.967855 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.971934 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.973300 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.973382 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:41.973586 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:41.974618 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0808 15:55:41.974775 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":457,"thread_start_us":245,"threads_started":1}
W20260808 15:55:41.975448 3121 leader_election.cc:334] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835): Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.975719 3683 leader_election.cc:288] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [CANDIDATE]: Term 2 election: Requested vote from peers 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835), c3259a5e0d32465faba0cfb4fe1e919f (127.27.254.60:43385)
I20260808 15:55:41.976140 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.955830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:35737 (local address 127.0.0.1:41438)
0808 15:55:41.964921 (+ 9091us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.964938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:41.965059 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:41.965521 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:41.965526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:41.965553 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:41.965830 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:41.965840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.968066 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.968078 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:41.971662 (+ 3584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:41.971684 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:41.974045 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:41.974054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:41.974308 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:41.974346 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:55:41.975676 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0808 15:55:41.975814 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9000,"mutex_wait_us":26,"thread_start_us":6903,"threads_started":1}
I20260808 15:55:41.976363 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.975773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:54974)
0808 15:55:41.976124 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.976210 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260808 15:55:41.976281 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a66c7b522c5e45a096157f6b545f2143" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c3259a5e0d32465faba0cfb4fe1e919f"
I20260808 15:55:41.976500 3147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:55:41.977475 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:41.976995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:55000)
0808 15:55:41.977184 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:41.977286 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 15:55:41.977813 2932 leader_election.cc:334] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835): Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
I20260808 15:55:41.979241 3147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66c7b522c5e45a096157f6b545f2143 in term 2.
I20260808 15:55:41.979601 2932 leader_election.cc:302] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a66c7b522c5e45a096157f6b545f2143, c3259a5e0d32465faba0cfb4fe1e919f; no voters: 993b6c092dd04040b740a1bfc66d0225
I20260808 15:55:41.979895 3683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:55:41.980109 2957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a66c7b522c5e45a096157f6b545f2143" is_pre_election: true
I20260808 15:55:41.980607 3683 raft_consensus.cc:686] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [term 2 LEADER]: Becoming Leader. State: Replica: a66c7b522c5e45a096157f6b545f2143, State: Running, Role: LEADER
I20260808 15:55:41.980926 3683 consensus_queue.cc:237] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } }
I20260808 15:55:41.983440 3693 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a66c7b522c5e45a096157f6b545f2143. Latest consensus state: current_term: 2 leader_uuid: "a66c7b522c5e45a096157f6b545f2143" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:41.983752 3693 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:41.984242 3120 leader_election.cc:302] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3259a5e0d32465faba0cfb4fe1e919f; no voters: 993b6c092dd04040b740a1bfc66d0225, a66c7b522c5e45a096157f6b545f2143
I20260808 15:55:41.984472 3695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:41.984606 3684 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:55:41.987964 3695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:41.988780 3695 catalog_manager.cc:1259] Loaded cluster ID: ed6059a0fcae44d1bae514f36c3a0794
I20260808 15:55:41.988881 3695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:41.990101 3695 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:41.990927 3695 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Loaded TSK: 0
I20260808 15:55:41.991750 3695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:55:42.034050 2947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:42.040871 2947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:42.044878 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.044447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:55022)
0808 15:55:42.044590 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.044696 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 15:55:42.045265 3147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [term 2 FOLLOWER]: Refusing update from remote peer a66c7b522c5e45a096157f6b545f2143: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 15:55:42.046164 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.045721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:55038)
0808 15:55:42.045893 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.046000 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 15:55:42.046120 3693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260808 15:55:42.046442 2932 consensus_peers.cc:487] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 -> Peer 993b6c092dd04040b740a1bfc66d0225 (127.27.254.62:46835): Couldn't send request to peer 993b6c092dd04040b740a1bfc66d0225. Status: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 15:55:42.048714 3684 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a66c7b522c5e45a096157f6b545f2143. Latest consensus state: current_term: 2 leader_uuid: "a66c7b522c5e45a096157f6b545f2143" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:42.048935 3684 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:55:42.050266 3683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a66c7b522c5e45a096157f6b545f2143" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:42.050593 3683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:42.051353 3684 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a66c7b522c5e45a096157f6b545f2143" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "993b6c092dd04040b740a1bfc66d0225" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46835 } } peers { permanent_uuid: "a66c7b522c5e45a096157f6b545f2143" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 35737 } attrs { promote: false } } peers { permanent_uuid: "c3259a5e0d32465faba0cfb4fe1e919f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 43385 } attrs { promote: false } } }
I20260808 15:55:42.051608 3684 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3259a5e0d32465faba0cfb4fe1e919f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:55:42.066586 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.059175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35825 (local address 127.0.0.1:59612)
0808 15:55:42.059901 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.059922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:42.060046 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:42.061376 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:42.061381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:42.061404 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:42.061701 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.061709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.063063 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.063069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.064500 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.064512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.065802 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.065809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.065976 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.066012 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:55:42.066095 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:55:42.066248 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595}
I20260808 15:55:42.067833 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.060433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.27.254.2:35825)
0808 15:55:42.061096 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.061100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.061116 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:55:42.061175 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.061179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.061243 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.061364 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:42.061844 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.062922 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.064662 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.065545 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.067192 (+ 1647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.067319 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.067371 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.067516 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":467,"threads_started":1}
I20260808 15:55:42.068732 3401 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1f9d7b4e7744f28584effec37351ed (DEFAULT_TABLE table=test-table [id=ab84c29ebc4e43ef85eb119d723bdd0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:42.069330 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1f9d7b4e7744f28584effec37351ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:42.072649 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.059521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:43015 (local address 127.0.0.1:55000)
0808 15:55:42.066846 (+ 7325us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.066863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:42.066992 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:42.067513 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:42.067518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:42.067540 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:42.067720 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.067729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.069195 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.069200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.070848 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.070859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.071911 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.071917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.072107 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.072130 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:55:42.072229 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 15:55:42.072360 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7225,"threads_started":1}
I20260808 15:55:42.072690 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.060370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.27.254.3:43015)
0808 15:55:42.060820 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.060824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.066953 (+ 6129us) server_negotiation.cc:407] Connection header received
0808 15:55:42.067138 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.067143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.067232 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.067322 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:42.068289 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.069051 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.070995 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.071756 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.071902 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.072009 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.072264 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.072394 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":267,"threads_started":1}
I20260808 15:55:42.074335 3543 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1f9d7b4e7744f28584effec37351ed (DEFAULT_TABLE table=test-table [id=ab84c29ebc4e43ef85eb119d723bdd0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:42.074781 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1f9d7b4e7744f28584effec37351ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:42.076920 3706 tablet_bootstrap.cc:492] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Bootstrap starting.
I20260808 15:55:42.078711 3706 tablet_bootstrap.cc:654] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:42.079877 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.058842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:40339 (local address 127.0.0.1:36302)
0808 15:55:42.059257 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.059277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:42.059417 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:42.074428 (+ 15011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:42.074435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:42.074464 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:42.074615 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.074625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.076306 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.076311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.077992 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.078004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.079159 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.079166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.079336 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.079370 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:55:42.079474 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:55:42.079602 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260808 15:55:42.079946 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.059674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.27.254.1:40339)
0808 15:55:42.074094 (+ 14420us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.074101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.074121 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:55:42.074188 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.074193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.074266 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.074377 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:42.075040 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.076145 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.078163 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.079007 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.079143 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.079255 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.079498 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.079639 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14299,"thread_start_us":14186,"threads_started":1}
I20260808 15:55:42.080729 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1f9d7b4e7744f28584effec37351ed (DEFAULT_TABLE table=test-table [id=ab84c29ebc4e43ef85eb119d723bdd0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:42.081173 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1f9d7b4e7744f28584effec37351ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:42.081802 3706 tablet_bootstrap.cc:492] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: No bootstrap required, opened a new log
I20260808 15:55:42.081947 3706 ts_tablet_manager.cc:1397] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 15:55:42.082716 3706 raft_consensus.cc:348] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.082944 3706 raft_consensus.cc:374] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:42.083034 3706 raft_consensus.cc:729] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620e184eeaf44ad19d7c4dd6140af409, State: Initialized, Role: FOLLOWER
I20260808 15:55:42.083298 3706 consensus_queue.cc:260] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.084069 3706 ts_tablet_manager.cc:1428] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:55:42.085294 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.084322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:33839)
0808 15:55:42.084930 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.085075 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":336,"threads_started":1}
I20260808 15:55:42.086935 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.086604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:53555)
0808 15:55:42.086743 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.086820 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:55:42.087577 3469 heartbeater.cc:503] Master 127.27.254.61:35737 was elected leader, sending a full tablet report...
I20260808 15:55:42.088723 3708 raft_consensus.cc:482] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:42.088891 3708 raft_consensus.cc:504] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.089287 3710 tablet_bootstrap.cc:492] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Bootstrap starting.
I20260808 15:55:42.089962 3708 leader_election.cc:288] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015), d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339)
I20260808 15:55:42.090341 3711 tablet_bootstrap.cc:492] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Bootstrap starting.
I20260808 15:55:42.090508 3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1f9d7b4e7744f28584effec37351ed" candidate_uuid: "620e184eeaf44ad19d7c4dd6140af409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72fbc8141f448ee82771422f4411e37" is_pre_election: true
I20260808 15:55:42.091127 3710 tablet_bootstrap.cc:654] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:42.091132 3562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1f9d7b4e7744f28584effec37351ed" candidate_uuid: "620e184eeaf44ad19d7c4dd6140af409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" is_pre_election: true
W20260808 15:55:42.091507 3357 leader_election.cc:341] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339): Illegal state: must be running to vote when last-logged opid is not known
W20260808 15:55:42.091910 3354 leader_election.cc:341] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015): Illegal state: must be running to vote when last-logged opid is not known
I20260808 15:55:42.092028 3354 leader_election.cc:302] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 620e184eeaf44ad19d7c4dd6140af409; no voters: d72fbc8141f448ee82771422f4411e37, e9f32ff8506f4806ab40be69e08ec0d0
I20260808 15:55:42.092233 3708 raft_consensus.cc:2726] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 15:55:42.092784 3711 tablet_bootstrap.cc:654] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:42.093230 3710 tablet_bootstrap.cc:492] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: No bootstrap required, opened a new log
I20260808 15:55:42.093362 3710 ts_tablet_manager.cc:1397] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260808 15:55:42.093992 3710 raft_consensus.cc:348] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.094167 3710 raft_consensus.cc:374] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:42.094290 3710 raft_consensus.cc:729] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72fbc8141f448ee82771422f4411e37, State: Initialized, Role: FOLLOWER
I20260808 15:55:42.094527 3710 consensus_queue.cc:260] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.094725 3711 tablet_bootstrap.cc:492] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: No bootstrap required, opened a new log
I20260808 15:55:42.094853 3711 ts_tablet_manager.cc:1397] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 15:55:42.095449 3711 raft_consensus.cc:348] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.095638 3711 raft_consensus.cc:374] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:42.095729 3711 raft_consensus.cc:729] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f32ff8506f4806ab40be69e08ec0d0, State: Initialized, Role: FOLLOWER
I20260808 15:55:42.095971 3711 consensus_queue.cc:260] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.096688 3711 ts_tablet_manager.cc:1428] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:55:42.098869 3710 ts_tablet_manager.cc:1428] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 15:55:42.100044 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.099033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:59795)
0808 15:55:42.099654 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.099796 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
I20260808 15:55:42.101621 3319 heartbeater.cc:503] Master 127.27.254.61:35737 was elected leader, sending a full tablet report...
I20260808 15:55:42.101840 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.101257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:37363)
0808 15:55:42.101414 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.101649 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 15:55:42.103093 3612 heartbeater.cc:503] Master 127.27.254.61:35737 was elected leader, sending a full tablet report...
I20260808 15:55:42.106181 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.097042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:46211)
0808 15:55:42.105774 (+ 8732us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.105920 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8622,"thread_start_us":8488,"threads_started":1}
I20260808 15:55:42.107017 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.106746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:46115)
0808 15:55:42.106864 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.106922 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 15:55:42.150156 3714 raft_consensus.cc:482] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:42.150388 3714 raft_consensus.cc:504] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.151990 3714 leader_election.cc:288] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015), 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825)
I20260808 15:55:42.159615 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.152418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:53109 (local address 127.27.254.3:43015)
0808 15:55:42.152594 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.152602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.152625 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:55:42.152843 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.152851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.152930 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.153019 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:42.153661 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.154654 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.157215 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.158476 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.158651 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.158723 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.159224 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.159348 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:55:42.159614 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.152527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:53247 (local address 127.27.254.2:35825)
0808 15:55:42.152755 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.152762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.153168 (+ 406us) server_negotiation.cc:407] Connection header received
0808 15:55:42.153302 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.153307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.153367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.153482 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:42.153963 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.154912 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.157215 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.158490 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.158652 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.158724 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.159224 (+ 500us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.159348 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:55:42.159869 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.152166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:43015 (local address 127.27.254.1:53109)
0808 15:55:42.152387 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.152407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:42.152585 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:42.153082 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:42.153089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:42.153124 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:42.153490 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.153502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.154779 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.154782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.157055 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.157070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.158708 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.158712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.159019 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.159084 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 15:55:42.159179 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:55:42.159342 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":133,"spinlock_wait_cycles":5248}
I20260808 15:55:42.159893 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.152347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35825 (local address 127.27.254.1:53247)
0808 15:55:42.153020 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.153038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:42.153171 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:42.153507 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:42.153511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:42.153550 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:42.153816 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.153823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.155031 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.155036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.157068 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:42.157077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.158702 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.158708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:42.159017 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.159084 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 15:55:42.159179 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:55:42.159342 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":194,"spinlock_wait_cycles":384,"thread_start_us":425,"threads_started":1}
I20260808 15:55:42.160631 3562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1f9d7b4e7744f28584effec37351ed" candidate_uuid: "d72fbc8141f448ee82771422f4411e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" is_pre_election: true
I20260808 15:55:42.160871 3562 raft_consensus.cc:2454] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d72fbc8141f448ee82771422f4411e37 in term 0.
I20260808 15:55:42.160722 3422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1f9d7b4e7744f28584effec37351ed" candidate_uuid: "d72fbc8141f448ee82771422f4411e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620e184eeaf44ad19d7c4dd6140af409" is_pre_election: true
I20260808 15:55:42.161151 3422 raft_consensus.cc:2454] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d72fbc8141f448ee82771422f4411e37 in term 0.
I20260808 15:55:42.161304 3204 leader_election.cc:302] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d72fbc8141f448ee82771422f4411e37, e9f32ff8506f4806ab40be69e08ec0d0; no voters:
I20260808 15:55:42.161775 3714 raft_consensus.cc:2781] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:42.161912 3714 raft_consensus.cc:482] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:42.162022 3714 raft_consensus.cc:3037] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:42.164309 3714 raft_consensus.cc:504] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.164861 3714 leader_election.cc:288] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [CANDIDATE]: Term 1 election: Requested vote from peers e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015), 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825)
I20260808 15:55:42.165355 3562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1f9d7b4e7744f28584effec37351ed" candidate_uuid: "d72fbc8141f448ee82771422f4411e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f32ff8506f4806ab40be69e08ec0d0"
I20260808 15:55:42.165369 3422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1f9d7b4e7744f28584effec37351ed" candidate_uuid: "d72fbc8141f448ee82771422f4411e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "620e184eeaf44ad19d7c4dd6140af409"
I20260808 15:55:42.165599 3422 raft_consensus.cc:3037] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:42.165619 3562 raft_consensus.cc:3037] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:42.167658 3562 raft_consensus.cc:2454] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d72fbc8141f448ee82771422f4411e37 in term 1.
I20260808 15:55:42.167675 3422 raft_consensus.cc:2454] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d72fbc8141f448ee82771422f4411e37 in term 1.
I20260808 15:55:42.168022 3204 leader_election.cc:302] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d72fbc8141f448ee82771422f4411e37, e9f32ff8506f4806ab40be69e08ec0d0; no voters:
I20260808 15:55:42.168336 3714 raft_consensus.cc:2781] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:42.168829 3714 raft_consensus.cc:686] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 1 LEADER]: Becoming Leader. State: Replica: d72fbc8141f448ee82771422f4411e37, State: Running, Role: LEADER
I20260808 15:55:42.169238 3714 consensus_queue.cc:237] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } }
I20260808 15:55:42.169955 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.169608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:55991)
0808 15:55:42.169742 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.169836 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 15:55:42.172814 2946 catalog_manager.cc:5495] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 reported cstate change: term changed from 0 to 1, leader changed from <none> to d72fbc8141f448ee82771422f4411e37 (127.27.254.1). New cstate: current_term: 1 leader_uuid: "d72fbc8141f448ee82771422f4411e37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d72fbc8141f448ee82771422f4411e37" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:42.190398 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.185500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.27.254.1:40339)
0808 15:55:42.185689 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.185693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.185730 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:55:42.185924 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.185927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.186004 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.186132 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:42.186418 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.187082 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.188323 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.188765 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.188858 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.189235 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:42.189900 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:42.190094 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.190141 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.190205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:55:42.194404 3562 raft_consensus.cc:1261] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Refusing update from remote peer d72fbc8141f448ee82771422f4411e37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:42.194824 3422 raft_consensus.cc:1261] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Refusing update from remote peer d72fbc8141f448ee82771422f4411e37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:42.195178 3719 consensus_queue.cc:1035] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f32ff8506f4806ab40be69e08ec0d0" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:42.195683 3714 consensus_queue.cc:1035] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "620e184eeaf44ad19d7c4dd6140af409" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:42.198233 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.197754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:39785)
0808 15:55:42.197910 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.198032 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 15:55:42.198808 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.198146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:57469)
0808 15:55:42.198407 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.198535 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260808 15:55:42.201473 3722 mvcc.cc:205] Tried to move back new op lower bound from 7316293804821422080 to 7316293804729520128. Current Snapshot: MvccSnapshot[applied={T|T < 7316293804821422080}]
I20260808 15:55:42.204046 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.203400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.2:48411)
0808 15:55:42.203697 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.203801 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260808 15:55:42.210925 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.210441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:52653)
0808 15:55:42.210657 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.210770 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 15:55:42.212694 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.212250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.3:59543)
0808 15:55:42.212408 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.212499 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:55:42.214605 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.214058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.27.254.1:39443)
0808 15:55:42.214262 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.214397 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:55:42.223424 2946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:42.223690 2946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:42.229063 2946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a66c7b522c5e45a096157f6b545f2143: Sending DeleteTablet for 3 replicas of tablet 8f1f9d7b4e7744f28584effec37351ed
I20260808 15:55:42.230492 3543 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1f9d7b4e7744f28584effec37351ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:42 UTC) from {username='oryx', principal='oryx/127.27.254.61@KRBTEST.COM'} at 127.0.0.1:55000
I20260808 15:55:42.230857 3401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1f9d7b4e7744f28584effec37351ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:42 UTC) from {username='oryx', principal='oryx/127.27.254.61@KRBTEST.COM'} at 127.0.0.1:59612
I20260808 15:55:42.230973 3251 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1f9d7b4e7744f28584effec37351ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:42 UTC) from {username='oryx', principal='oryx/127.27.254.61@KRBTEST.COM'} at 127.0.0.1:36302
I20260808 15:55:42.231185 3732 tablet_replica.cc:330] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: stopping tablet replica
I20260808 15:55:42.231398 3732 raft_consensus.cc:2229] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:42.231769 3732 raft_consensus.cc:2258] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:42.232767 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 3113
I20260808 15:55:42.242095 3732 ts_tablet_manager.cc:1905] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:42.252317 3732 ts_tablet_manager.cc:1918] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:42.253932 3733 tablet_replica.cc:330] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: stopping tablet replica
I20260808 15:55:42.254283 3733 raft_consensus.cc:2229] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:42.254698 3733 raft_consensus.cc:2258] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:42.256140 3733 ts_tablet_manager.cc:1905] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:42.256668 3734 tablet_replica.cc:330] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: stopping tablet replica
I20260808 15:55:42.256870 3734 raft_consensus.cc:2229] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:42.257568 3734 raft_consensus.cc:2258] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:42.258724 3734 ts_tablet_manager.cc:1905] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:42.260100 3732 log.cc:1196] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/8f1f9d7b4e7744f28584effec37351ed
I20260808 15:55:42.260612 3732 ts_tablet_manager.cc:1939] T 8f1f9d7b4e7744f28584effec37351ed P e9f32ff8506f4806ab40be69e08ec0d0: Deleting consensus metadata
I20260808 15:55:42.261879 2931 catalog_manager.cc:4841] TS e9f32ff8506f4806ab40be69e08ec0d0 (127.27.254.3:43015): tablet 8f1f9d7b4e7744f28584effec37351ed (table test-table [id=ab84c29ebc4e43ef85eb119d723bdd0e]) successfully deleted
I20260808 15:55:42.265933 3734 ts_tablet_manager.cc:1918] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:42.266067 3734 log.cc:1196] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/8f1f9d7b4e7744f28584effec37351ed
I20260808 15:55:42.266541 3734 ts_tablet_manager.cc:1939] T 8f1f9d7b4e7744f28584effec37351ed P d72fbc8141f448ee82771422f4411e37: Deleting consensus metadata
I20260808 15:55:42.267555 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.233689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.27.254.1:40339)
0808 15:55:42.233976 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.233982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.246102 (+ 12120us) server_negotiation.cc:407] Connection header received
0808 15:55:42.246312 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.246322 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.246443 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.246611 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:42.248699 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.250407 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.252183 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.253159 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.253323 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.257493 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:42.257509 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:42.257528 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:42.257572 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:42.262361 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:42.265022 (+ 2661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:42.265031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:42.265034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:42.265262 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:42.265723 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:42.265729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:42.265732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:42.266357 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:42.266766 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.266956 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.267179 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260808 15:55:42.267674 2934 catalog_manager.cc:4841] TS d72fbc8141f448ee82771422f4411e37 (127.27.254.1:40339): tablet 8f1f9d7b4e7744f28584effec37351ed (table test-table [id=ab84c29ebc4e43ef85eb119d723bdd0e]) successfully deleted
I20260808 15:55:42.268525 3312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:42.268512 3733 ts_tablet_manager.cc:1918] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:42.268656 3733 log.cc:1196] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/8f1f9d7b4e7744f28584effec37351ed
I20260808 15:55:42.269161 3733 ts_tablet_manager.cc:1939] T 8f1f9d7b4e7744f28584effec37351ed P 620e184eeaf44ad19d7c4dd6140af409: Deleting consensus metadata
I20260808 15:55:42.270354 2933 catalog_manager.cc:4841] TS 620e184eeaf44ad19d7c4dd6140af409 (127.27.254.2:35825): tablet 8f1f9d7b4e7744f28584effec37351ed (table test-table [id=ab84c29ebc4e43ef85eb119d723bdd0e]) successfully deleted
I20260808 15:55:42.367806 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 3113
I20260808 15:55:42.386404 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 3329
I20260808 15:55:42.399641 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.387171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.27.254.2:35825)
0808 15:55:42.387354 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.387362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.387448 (+ 86us) server_negotiation.cc:407] Connection header received
0808 15:55:42.387662 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.387668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.387744 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.387857 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:42.389132 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.389984 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.391264 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.391669 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.391801 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.394318 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:42.394340 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:42.394356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:42.394388 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:42.396919 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:42.397616 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:42.397625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:42.397635 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:42.397748 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:42.398121 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:42.398128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:42.398131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:42.398482 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:42.398808 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.399094 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.399315 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:55:42.400314 3462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:42.519901 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 3329
I20260808 15:55:42.529820 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.529384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:55048)
0808 15:55:42.529596 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.529684 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:55:42.530797 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.530462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46835 (local address 127.0.0.1:55054)
0808 15:55:42.530602 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:42.530681 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:46835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 15:55:42.538465 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 3480
I20260808 15:55:42.551978 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:42.539201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.27.254.3:43015)
0808 15:55:42.539380 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:42.539387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:42.539479 (+ 92us) server_negotiation.cc:407] Connection header received
0808 15:55:42.539690 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:42.539696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:42.539776 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:42.539880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:42.541207 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.542067 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.543300 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:42.543707 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:42.543842 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:42.546639 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:42.546654 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:42.546670 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:42.546716 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:42.549368 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:42.550017 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:42.550025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:42.550027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:42.550172 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:42.550492 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:42.550495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:42.550497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:42.550825 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:42.551140 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:42.551442 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 15:55:42.551656 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:55:42.552600 3605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:42.644459 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 3480
I20260808 15:55:42.662599 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2810
I20260808 15:55:42.663791 2977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:42.784945 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2810
I20260808 15:55:42.803521 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 2911
I20260808 15:55:42.804595 3167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:42.922286 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 2911
2026-08-08T15:55:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9352 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:43 dist-test-slave-srgv krb5kdc[3743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:43 dist-test-slave-srgv krb5kdc[3743](info): set up 2 sockets
Aug 08 15:55:43 dist-test-slave-srgv krb5kdc[3743](info): commencing operation
krb5kdc: starting...
W20260808 15:55:44.024292 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:44 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab.
Entry for principal oryx/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab.
I20260808 15:55:44.228826 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33183
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab
--principal=oryx/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:44.549373 3756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:44.549712 3756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:44.549876 3756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:44.556828 3756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:44.556974 3756 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:55:44.557077 3756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:44.557189 3756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:44.557283 3756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:44.557370 3756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:44.566560 3756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33183
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.62.keytab
--principal=oryx/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:44.567818 3756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:44.569132 3756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:44.580415 3761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:44.580368 3764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:44.580343 3762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:44.581470 3756 server_base.cc:1034] running on GCE node
I20260808 15:55:44.582182 3756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:44.583498 3756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:44.584717 3756 hybrid_clock.cc:648] HybridClock initialized: now 1786204544584646 us; error 49 us; skew 500 ppm
I20260808 15:55:44.585040 3756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:44 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204544, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:44.588105 3756 init.cc:377] Logged in from keytab as oryx/127.27.254.62@KRBTEST.COM (short username oryx)
I20260808 15:55:44.589555 3756 webserver.cc:460] Webserver started at http://127.27.254.62:37119/ using document root <none> and password file <none>
I20260808 15:55:44.590242 3756 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:44.590355 3756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:44.590691 3756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:44.593569 3756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "c54ac9f031a743d98616435a98a64f64"
format_stamp: "Formatted at 2026-08-08 15:55:44 on dist-test-slave-srgv"
server_key: "957cbda8ca396a4ba364da59c47431ce"
server_key_iv: "d41785d88b098c8e6322571ce1994b11"
server_key_version: "encryptionkey@0"
I20260808 15:55:44.594370 3756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "c54ac9f031a743d98616435a98a64f64"
format_stamp: "Formatted at 2026-08-08 15:55:44 on dist-test-slave-srgv"
server_key: "957cbda8ca396a4ba364da59c47431ce"
server_key_iv: "d41785d88b098c8e6322571ce1994b11"
server_key_version: "encryptionkey@0"
I20260808 15:55:44.599282 3756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:55:44.602571 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:44.603936 3756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:55:44.604108 3756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "c54ac9f031a743d98616435a98a64f64"
format_stamp: "Formatted at 2026-08-08 15:55:44 on dist-test-slave-srgv"
server_key: "957cbda8ca396a4ba364da59c47431ce"
server_key_iv: "d41785d88b098c8e6322571ce1994b11"
server_key_version: "encryptionkey@0"
I20260808 15:55:44.604321 3756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:44.623377 3756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:44.627575 3756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:44.627862 3756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:44.649274 3756 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33183
I20260808 15:55:44.649299 3832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33183 every 8 connection(s)
I20260808 15:55:44.650701 3756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:44.654349 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:44.657591 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 3756
I20260808 15:55:44.657789 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:44.658444 28664 external_mini_cluster.cc:1428] Setting key bf569782e0134061894ef073ee5e1be4
I20260808 15:55:44.667269 3833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Bootstrap starting.
I20260808 15:55:44.670783 3833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:44.671779 3833 log.cc:824] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:55:44 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:44.674618 3833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: No bootstrap required, opened a new log
I20260808 15:55:44.680938 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:44.660388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.27.254.62:33183)
0808 15:55:44.661204 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:44.661218 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:44.661267 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:55:44.663218 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:44.663249 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:44.663739 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:44.664181 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:44.665369 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:44.667273 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:44.668520 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:44.669584 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:44.669753 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:44.673898 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:44.673905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:44.673918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:44.673951 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:44.676959 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:44.677733 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:44.677740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:44.677745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:44.677903 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:44.678285 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:44.678290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:44.678293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:44.678818 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:44.679162 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:44.679585 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 15:55:44.679964 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":328,"threads_started":1}
I20260808 15:55:44.682613 3833 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54ac9f031a743d98616435a98a64f64" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33183 } }
I20260808 15:55:44.683036 3833 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:44.683168 3833 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54ac9f031a743d98616435a98a64f64, State: Initialized, Role: FOLLOWER
I20260808 15:55:44.683794 3833 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54ac9f031a743d98616435a98a64f64" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33183 } }
I20260808 15:55:44.684075 3833 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:44.684207 3833 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:44.684319 3833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:44.686385 3833 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54ac9f031a743d98616435a98a64f64" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33183 } }
I20260808 15:55:44.686800 3833 leader_election.cc:302] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54ac9f031a743d98616435a98a64f64; no voters:
I20260808 15:55:44.687323 3833 leader_election.cc:288] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:44.687433 3838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:44.688658 3838 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [term 1 LEADER]: Becoming Leader. State: Replica: c54ac9f031a743d98616435a98a64f64, State: Running, Role: LEADER
I20260808 15:55:44.689138 3838 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54ac9f031a743d98616435a98a64f64" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33183 } }
I20260808 15:55:44.689345 3833 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:44.691910 3841 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54ac9f031a743d98616435a98a64f64. Latest consensus state: current_term: 1 leader_uuid: "c54ac9f031a743d98616435a98a64f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54ac9f031a743d98616435a98a64f64" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33183 } } }
I20260808 15:55:44.692337 3841 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:44.692610 3840 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54ac9f031a743d98616435a98a64f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54ac9f031a743d98616435a98a64f64" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33183 } } }
I20260808 15:55:44.692904 3840 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:44.694519 3845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:44.699909 3845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:44.709257 3845 catalog_manager.cc:1347] Generated new cluster ID: 9695f9274cf24309a7b2f1b96eed8241
I20260808 15:55:44.709367 3845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:44.730022 3845 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:44.731365 3845 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:44.739118 3845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Generated new TSK 0
I20260808 15:55:44.740142 3845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab.
Entry for principal oryx/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab.
I20260808 15:55:44.792431 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33183
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:45.114447 3860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:45.114696 3860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:45.114851 3860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:45.121466 3860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:45.121574 3860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:45.121737 3860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:55:45.121851 3860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:45.130779 3860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:45.131995 3860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:45.133190 3860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:45.145994 3868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:45.146066 3866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:45.146044 3865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:45.146952 3860 server_base.cc:1034] running on GCE node
I20260808 15:55:45.147552 3860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:45.148455 3860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:45.149670 3860 hybrid_clock.cc:648] HybridClock initialized: now 1786204545149613 us; error 37 us; skew 500 ppm
I20260808 15:55:45.149957 3860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:45 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204545, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:45.153098 3860 init.cc:377] Logged in from keytab as oryx/127.27.254.1@KRBTEST.COM (short username oryx)
I20260808 15:55:45.154505 3860 webserver.cc:460] Webserver started at http://127.27.254.1:34027/ using document root <none> and password file <none>
I20260808 15:55:45.155112 3860 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:45.155198 3860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:45.155455 3860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:45.158218 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "9a56e55d31744ee7955c1c1b124a8a35"
format_stamp: "Formatted at 2026-08-08 15:55:45 on dist-test-slave-srgv"
server_key: "48e93c269790bb339ee02e8dd24b89c5"
server_key_iv: "4ebf83b7f9a544296a8d6e8ee0b80b6d"
server_key_version: "encryptionkey@0"
I20260808 15:55:45.158934 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "9a56e55d31744ee7955c1c1b124a8a35"
format_stamp: "Formatted at 2026-08-08 15:55:45 on dist-test-slave-srgv"
server_key: "48e93c269790bb339ee02e8dd24b89c5"
server_key_iv: "4ebf83b7f9a544296a8d6e8ee0b80b6d"
server_key_version: "encryptionkey@0"
I20260808 15:55:45.163714 3860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:55:45.166966 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:45.168232 3860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:55:45.168397 3860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "9a56e55d31744ee7955c1c1b124a8a35"
format_stamp: "Formatted at 2026-08-08 15:55:45 on dist-test-slave-srgv"
server_key: "48e93c269790bb339ee02e8dd24b89c5"
server_key_iv: "4ebf83b7f9a544296a8d6e8ee0b80b6d"
server_key_version: "encryptionkey@0"
I20260808 15:55:45.168614 3860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:45.192111 3860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:45.195984 3860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:45.196275 3860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:45.198921 3860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:45.199025 3860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:45.199216 3860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:45.199290 3860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:55:45 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204545, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:45.224078 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.199272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:57875 (local address 127.27.254.62:33183)
0808 15:55:45.199724 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:45.199731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:45.200987 (+ 1256us) server_negotiation.cc:407] Connection header received
0808 15:55:45.203182 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:45.203200 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:45.203310 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:45.203458 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:45.205509 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.206819 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.209210 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.209792 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.209930 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.215519 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:45.215527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:45.215535 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:45.215589 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:45.219508 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.220688 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.220694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.220697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.220872 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.221581 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.221589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.221594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.222018 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:45.222201 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:45.223320 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0808 15:55:45.223598 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":231,"threads_started":1}
I20260808 15:55:45.225666 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.199943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.27.254.1:57875)
0808 15:55:45.200812 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:45.200864 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:45.202800 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:45.203711 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:45.203721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:45.204451 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:45.205281 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.205324 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.206952 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.206957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.209074 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.209084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.210079 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.210088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.210412 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.211175 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:45.211230 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:45.215188 (+ 3958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:45.219688 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.219697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.219720 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.220519 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.221003 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.221008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.221012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.221336 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.222166 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:45.222178 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:45.222803 (+ 625us) client_negotiation.cc:769] Sending connection context
0808 15:55:45.223180 (+ 377us) client_negotiation.cc:240] Negotiation successful
0808 15:55:45.223895 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":395,"threads_started":1}
I20260808 15:55:45.239974 3860 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:46733
I20260808 15:55:45.239989 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:46733 every 8 connection(s)
I20260808 15:55:45.241513 3860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:45.243546 3998 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33183
I20260808 15:55:45.244304 3998 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:45.244946 3998 heartbeater.cc:511] Master 127.27.254.62:33183 requested a full tablet report, sending...
I20260808 15:55:45.246332 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 3860
I20260808 15:55:45.246534 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:45.247161 3788 ts_manager.cc:194] Registered new tserver with Master: 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733)
I20260808 15:55:45.247354 28664 external_mini_cluster.cc:1428] Setting key 62c3160cbdba9119b4ca04a7f861a3ef
I20260808 15:55:45.248253 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.234358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.27.254.1:42989)
0808 15:55:45.235166 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:45.235189 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:45.235351 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:45.235825 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:45.235829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:45.236286 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:45.236883 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.236895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.237972 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.237977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.239306 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.239319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.240359 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.240364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.240489 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.241108 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:45.241124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:45.242169 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:45.245337 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.245343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.245346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.245848 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.246253 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.246259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.246263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.246426 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.247208 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:45.247214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:45.247403 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 15:55:45.247558 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 15:55:45.247836 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":430,"threads_started":1}
I20260808 15:55:45.249033 3788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:57875
I20260808 15:55:45.249256 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.234196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:42989 (local address 127.27.254.62:33183)
0808 15:55:45.234386 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:45.234390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:45.235288 (+ 898us) server_negotiation.cc:407] Connection header received
0808 15:55:45.235506 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:45.235514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:45.235644 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:45.235789 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:45.237037 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.237828 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.239529 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.240163 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.240293 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.242396 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:45.242402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:45.242406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:45.242441 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:45.245217 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.245967 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.245974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.245977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.246143 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.246578 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.246584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.246588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.246976 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:45.247126 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:45.248582 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0808 15:55:45.248798 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for oryx/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab.
Entry for principal oryx/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab.
I20260808 15:55:45.292443 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33183
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:45.607923 4002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:45.608248 4002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:45.608449 4002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:45.615706 4002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:45.615854 4002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:45.616014 4002 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:55:45.616149 4002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:45.625916 4002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:45.627172 4002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:45.628507 4002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:45.641911 4007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:45.641937 4010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:45.641871 4008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:45.642881 4002 server_base.cc:1034] running on GCE node
I20260808 15:55:45.643505 4002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:45.644315 4002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:45.645522 4002 hybrid_clock.cc:648] HybridClock initialized: now 1786204545645474 us; error 28 us; skew 500 ppm
I20260808 15:55:45.645771 4002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:45 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204545, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:45.648885 4002 init.cc:377] Logged in from keytab as oryx/127.27.254.2@KRBTEST.COM (short username oryx)
I20260808 15:55:45.650594 4002 webserver.cc:460] Webserver started at http://127.27.254.2:41887/ using document root <none> and password file <none>
I20260808 15:55:45.651434 4002 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:45.651543 4002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:45.651875 4002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:45.655704 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "671e9de0011c464e8b40942eec847f5a"
format_stamp: "Formatted at 2026-08-08 15:55:45 on dist-test-slave-srgv"
server_key: "ef3e3b66caaa961c66b4cff890d3059f"
server_key_iv: "4df9b9a10802ab177a482b30844af6f6"
server_key_version: "encryptionkey@0"
I20260808 15:55:45.656629 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "671e9de0011c464e8b40942eec847f5a"
format_stamp: "Formatted at 2026-08-08 15:55:45 on dist-test-slave-srgv"
server_key: "ef3e3b66caaa961c66b4cff890d3059f"
server_key_iv: "4df9b9a10802ab177a482b30844af6f6"
server_key_version: "encryptionkey@0"
I20260808 15:55:45.662765 4002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 15:55:45.666955 4017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:45.668406 4002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:55:45.668548 4002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "671e9de0011c464e8b40942eec847f5a"
format_stamp: "Formatted at 2026-08-08 15:55:45 on dist-test-slave-srgv"
server_key: "ef3e3b66caaa961c66b4cff890d3059f"
server_key_iv: "4df9b9a10802ab177a482b30844af6f6"
server_key_version: "encryptionkey@0"
I20260808 15:55:45.668761 4002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:45.692490 4002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:45.696336 4002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:45.696652 4002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:45.699155 4002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:45.699290 4002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:45.699455 4002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:45.699542 4002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:55:45 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204545, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:45.723071 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.699898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:54373 (local address 127.27.254.62:33183)
0808 15:55:45.700117 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:45.700126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:45.701606 (+ 1480us) server_negotiation.cc:407] Connection header received
0808 15:55:45.703648 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:45.703655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:45.703740 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:45.703857 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:45.706023 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.707353 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.709943 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.710583 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.710725 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.715566 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:45.715573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:45.715580 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:45.715627 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:45.719274 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.720341 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.720345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.720347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.720508 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.721091 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.721096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.721099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.721561 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:45.721753 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:45.722505 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 15:55:45.722722 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:55:45.724442 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.700493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.27.254.2:54373)
0808 15:55:45.701434 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:45.701483 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:45.703340 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:45.704097 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:45.704105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:45.704850 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:45.705799 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.705831 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.707558 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.707567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.709784 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.709796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.710916 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.710927 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.711280 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.712069 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:45.712112 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:45.715278 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:45.719466 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.719477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.719494 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.720176 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.720670 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.720677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.720683 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.720963 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.721726 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:45.721738 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:45.722129 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 15:55:45.722421 (+ 292us) client_negotiation.cc:240] Negotiation successful
0808 15:55:45.722979 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":430,"threads_started":1}
I20260808 15:55:45.745375 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.731110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:37651 (local address 127.27.254.62:33183)
0808 15:55:45.731304 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:45.731309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:45.731438 (+ 129us) server_negotiation.cc:407] Connection header received
0808 15:55:45.731658 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:45.731662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:45.731715 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:45.731812 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:45.732701 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.733399 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.735549 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.736138 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.736264 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.738452 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:45.738458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:45.738463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:45.738502 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:45.742677 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.743548 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.743555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.743559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.743698 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:45.744033 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:45.744040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:45.744043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:45.744331 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:45.744474 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:45.744870 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 15:55:45.745057 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:55:45.745496 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:45.730728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.27.254.2:37651)
0808 15:55:45.731336 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:45.731354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:45.731506 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:45.731834 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:45.731838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:45.732266 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:45.732540 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.732548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.733608 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.733614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.735412 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:45.735430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:45.736330 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:45.736337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:45.736492 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:45.737082 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:45.737100 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:45.738238 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:45.742822 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.742826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.742828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.743411 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.743815 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:45.743820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:45.743823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:45.743934 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:45.744441 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:45.744445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:45.744606 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 15:55:45.744740 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 15:55:45.745049 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":439,"threads_started":1}
I20260808 15:55:45.751384 4002 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:35329
I20260808 15:55:45.751406 4136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:35329 every 8 connection(s)
I20260808 15:55:45.752525 4002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:45.754366 4137 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33183
I20260808 15:55:45.754577 4137 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:45.755147 4137 heartbeater.cc:511] Master 127.27.254.62:33183 requested a full tablet report, sending...
I20260808 15:55:45.756340 3787 ts_manager.cc:194] Registered new tserver with Master: 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:45.757146 3787 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:54373
I20260808 15:55:45.761364 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4002
I20260808 15:55:45.761598 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:45.762068 28664 external_mini_cluster.cc:1428] Setting key c514114ce080bc364c9ee5d2baf92fb5
WARNING: no policy specified for oryx/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab.
Entry for principal oryx/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab.
I20260808 15:55:45.808339 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33183
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:46.134372 4141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:46.134678 4141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:46.134874 4141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:46.142578 4141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:46.142729 4141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:46.142872 4141 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:55:46.143013 4141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:46.152482 4141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.254.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:46.153862 4141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:46.155167 4141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:46.168318 4147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:46.168375 4149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:46.168246 4146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:46.169210 4141 server_base.cc:1034] running on GCE node
I20260808 15:55:46.169991 4141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:46.170908 4141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:46.172142 4141 hybrid_clock.cc:648] HybridClock initialized: now 1786204546172065 us; error 53 us; skew 500 ppm
I20260808 15:55:46.172439 4141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:46 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204546, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:46.175782 4141 init.cc:377] Logged in from keytab as oryx/127.27.254.3@KRBTEST.COM (short username oryx)
I20260808 15:55:46.177402 4141 webserver.cc:460] Webserver started at http://127.27.254.3:35659/ using document root <none> and password file <none>
I20260808 15:55:46.178141 4141 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:46.178227 4141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:46.178483 4141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:46.181191 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "19f5277bd5934d64852f6b3e4d6ed19e"
format_stamp: "Formatted at 2026-08-08 15:55:46 on dist-test-slave-srgv"
server_key: "a3e17a1ce6c4db58ef07aff987f7434f"
server_key_iv: "0301953d67676cc0d9e423f09eec6ab4"
server_key_version: "encryptionkey@0"
I20260808 15:55:46.181996 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "19f5277bd5934d64852f6b3e4d6ed19e"
format_stamp: "Formatted at 2026-08-08 15:55:46 on dist-test-slave-srgv"
server_key: "a3e17a1ce6c4db58ef07aff987f7434f"
server_key_iv: "0301953d67676cc0d9e423f09eec6ab4"
server_key_version: "encryptionkey@0"
I20260808 15:55:46.186767 4141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:55:46.189955 4156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:46.191252 4141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:55:46.191380 4141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "19f5277bd5934d64852f6b3e4d6ed19e"
format_stamp: "Formatted at 2026-08-08 15:55:46 on dist-test-slave-srgv"
server_key: "a3e17a1ce6c4db58ef07aff987f7434f"
server_key_iv: "0301953d67676cc0d9e423f09eec6ab4"
server_key_version: "encryptionkey@0"
I20260808 15:55:46.191543 4141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:46.207656 4141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:46.211552 4141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:46.211772 4141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:46.214305 4141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:46.214388 4141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:46.214576 4141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:46.214658 4141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:55:46 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204546, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:46.236960 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:46.214719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:58041 (local address 127.27.254.62:33183)
0808 15:55:46.214908 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:46.214914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:46.216335 (+ 1421us) server_negotiation.cc:407] Connection header received
0808 15:55:46.218370 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:46.218375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:46.218474 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:46.218607 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:46.220671 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.221987 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.224337 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.224886 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.225018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:46.229691 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:46.229699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:46.229704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:46.229750 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:46.232869 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:46.233956 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:46.233963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:46.233969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:46.234172 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:46.234759 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:46.234765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:46.234771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:46.235195 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:46.235359 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:46.236361 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0808 15:55:46.236592 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:55:46.238080 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:46.215247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.27.254.3:58041)
0808 15:55:46.216133 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:46.216179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:46.218093 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:46.218877 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:46.218887 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:46.219668 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:46.220418 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:46.220439 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.222171 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.222178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:46.224174 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:46.224185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.225148 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.225159 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:46.225338 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:46.226064 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:46.226118 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:46.229438 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:46.233012 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:46.233025 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:46.233045 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:46.233800 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:46.234319 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:46.234324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:46.234328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:46.234602 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:46.235369 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:46.235380 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:46.235818 (+ 438us) client_negotiation.cc:769] Sending connection context
0808 15:55:46.236222 (+ 404us) client_negotiation.cc:240] Negotiation successful
0808 15:55:46.236754 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":413,"threads_started":1}
I20260808 15:55:46.257082 3998 heartbeater.cc:503] Master 127.27.254.62:33183 was elected leader, sending a full tablet report...
I20260808 15:55:46.260208 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:46.244936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.27.254.3:56531)
0808 15:55:46.245802 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:46.245826 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:46.245980 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:46.246300 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:46.246308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:46.247006 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:46.247446 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:46.247460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.248436 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.248444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:46.250448 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:46.250469 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.251379 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.251389 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:46.251597 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:46.252414 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:46.252439 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:46.253818 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:46.256650 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:46.256659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:46.256665 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:46.257329 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:46.257895 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:46.257902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:46.257908 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:46.258089 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:46.258915 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:46.258921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:46.259130 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 15:55:46.259284 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 15:55:46.259679 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":415,"threads_started":1}
I20260808 15:55:46.260327 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:46.244927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:56531 (local address 127.27.254.62:33183)
0808 15:55:46.245131 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:46.245139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:46.245931 (+ 792us) server_negotiation.cc:407] Connection header received
0808 15:55:46.246092 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:46.246096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:46.246161 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:46.246245 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:46.247589 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.248291 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.250646 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.251191 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.251339 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:46.254024 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:46.254032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:46.254037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:46.254071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:46.256524 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:46.257547 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:46.257556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:46.257562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:46.257742 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:46.258262 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:46.258266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:46.258269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:46.258677 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:46.259604 (+ 927us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:46.259757 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 15:55:46.259975 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:55:46.273485 4141 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:40621
I20260808 15:55:46.273507 4275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:40621 every 8 connection(s)
I20260808 15:55:46.274714 4141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:46.276643 4276 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33183
I20260808 15:55:46.276878 4276 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:46.277616 4276 heartbeater.cc:511] Master 127.27.254.62:33183 requested a full tablet report, sending...
I20260808 15:55:46.278887 3787 ts_manager.cc:194] Registered new tserver with Master: 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621)
I20260808 15:55:46.279450 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4141
I20260808 15:55:46.279629 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:46.280069 3787 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:58041
I20260808 15:55:46.280170 28664 external_mini_cluster.cc:1428] Setting key 89cb5036cceef172c52d85d3addd6965
I20260808 15:55:46.283093 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:55:46 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:55:46 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:46 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:46.759385 4137 heartbeater.cc:503] Master 127.27.254.62:33183 was elected leader, sending a full tablet report...
I20260808 15:55:47.283015 4276 heartbeater.cc:503] Master 127.27.254.62:33183 was elected leader, sending a full tablet report...
W20260808 15:55:47.414674 4284 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.089s user 0.007s sys 0.007s
W20260808 15:55:47.415858 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:46.320316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.27.254.62:33183)
0808 15:55:46.320550 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:46.320557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:46.320704 (+ 147us) server_negotiation.cc:407] Connection header received
0808 15:55:46.320858 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:46.320862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:46.320926 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:46.321010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:46.322354 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.323094 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.324300 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:46.324677 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:46.324769 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.415622 (+1090853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45336: BlockingRecv error: recv got EOF from 127.0.0.1:45336 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 08 15:55:47 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:47.436375 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.419824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.27.254.62:33183)
0808 15:55:47.419995 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.419999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.420346 (+ 347us) server_negotiation.cc:407] Connection header received
0808 15:55:47.420497 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.420501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.420586 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.420683 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:47.422035 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.422752 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.423949 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.424371 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.424458 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.431966 (+ 7508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:47.431971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:47.431975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:47.432009 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:47.434370 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:47.434928 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:47.434931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:47.434933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:47.435058 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:47.435349 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:47.435352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:47.435354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:47.435608 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:47.435834 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.435991 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.436137 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:55:47.440421 3787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:47.447160 3787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:47.474009 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.458718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.27.254.2:35329)
0808 15:55:47.461044 (+ 2326us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.461057 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.461093 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:55:47.461208 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.461220 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.461554 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.461785 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:47.462979 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.465049 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.468938 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.471036 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.472382 (+ 1346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.473429 (+ 1047us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.473505 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.473666 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2107,"thread_start_us":241,"threads_started":1}
I20260808 15:55:47.474519 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.458524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35329 (local address 127.0.0.1:37334)
0808 15:55:47.460286 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:47.460348 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:47.460551 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:47.462078 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:47.462090 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:47.462129 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:47.462703 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.462720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.465307 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.465315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.468699 (+ 3384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.468721 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.472009 (+ 3288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.472016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.472676 (+ 660us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.472736 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 15:55:47.472853 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 15:55:47.473841 (+ 988us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"thread_start_us":1273,"threads_started":1}
I20260808 15:55:47.478587 4067 tablet_service.cc:1467] Processing CreateTablet for tablet 321c39f952f54c39bf12409a8955d4ff (DEFAULT_TABLE table=test-table [id=9d76b778043c47aaba5e5b158b0aa4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:47.480036 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321c39f952f54c39bf12409a8955d4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:47.480818 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.458685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:46733 (local address 127.0.0.1:33220)
0808 15:55:47.472082 (+ 13397us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:47.472106 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:47.472366 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:47.473140 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:47.473145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:47.473172 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:47.474606 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.474621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.476319 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.476323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.478163 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.478178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.479946 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.479952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.480129 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.480161 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:55:47.480242 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 15:55:47.480362 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13260,"mutex_wait_us":29,"thread_start_us":13086,"threads_started":1}
I20260808 15:55:47.481612 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.461382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:40621 (local address 127.0.0.1:37186)
0808 15:55:47.473492 (+ 12110us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:47.473510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:47.473625 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:47.474340 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:47.474347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:47.474382 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:47.474893 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.474907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.476490 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.476495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.478645 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.478659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.480745 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.480750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.480921 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.480956 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:55:47.481042 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 15:55:47.481226 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11740,"mutex_wait_us":1,"thread_start_us":11366,"threads_started":1}
I20260808 15:55:47.481786 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.459295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.27.254.1:46733)
0808 15:55:47.471722 (+ 12427us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.471734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.472295 (+ 561us) server_negotiation.cc:407] Connection header received
0808 15:55:47.472597 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.472608 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.472911 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.473147 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:47.474814 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.476116 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.478322 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.479740 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.480960 (+ 1220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.481196 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.481263 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.481457 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12235,"thread_start_us":12086,"threads_started":1}
I20260808 15:55:47.482002 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.461013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.27.254.3:40621)
0808 15:55:47.472855 (+ 11842us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.472865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.473661 (+ 796us) server_negotiation.cc:407] Connection header received
0808 15:55:47.473874 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.473882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.474095 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.474897 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:47.475111 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.476316 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.478824 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.480258 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.480388 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.480587 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.481550 (+ 963us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.481705 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11589,"thread_start_us":11381,"threads_started":1}
I20260808 15:55:47.485260 3926 tablet_service.cc:1467] Processing CreateTablet for tablet 321c39f952f54c39bf12409a8955d4ff (DEFAULT_TABLE table=test-table [id=9d76b778043c47aaba5e5b158b0aa4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:47.485495 4206 tablet_service.cc:1467] Processing CreateTablet for tablet 321c39f952f54c39bf12409a8955d4ff (DEFAULT_TABLE table=test-table [id=9d76b778043c47aaba5e5b158b0aa4f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:47.486521 4206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321c39f952f54c39bf12409a8955d4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:47.486624 3926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321c39f952f54c39bf12409a8955d4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:47.496096 4301 tablet_bootstrap.cc:492] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Bootstrap starting.
I20260808 15:55:47.497728 4302 tablet_bootstrap.cc:492] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Bootstrap starting.
I20260808 15:55:47.499504 4301 tablet_bootstrap.cc:654] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:47.500247 4303 tablet_bootstrap.cc:492] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Bootstrap starting.
I20260808 15:55:47.500816 4301 log.cc:824] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:47.500849 4302 tablet_bootstrap.cc:654] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:47.502130 4302 log.cc:824] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:47.503597 4303 tablet_bootstrap.cc:654] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:47.504179 4301 tablet_bootstrap.cc:492] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: No bootstrap required, opened a new log
I20260808 15:55:47.504446 4301 ts_tablet_manager.cc:1397] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260808 15:55:47.504722 4303 log.cc:824] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:47.505316 4302 tablet_bootstrap.cc:492] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: No bootstrap required, opened a new log
I20260808 15:55:47.505601 4302 ts_tablet_manager.cc:1397] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:55:47.507553 4303 tablet_bootstrap.cc:492] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: No bootstrap required, opened a new log
I20260808 15:55:47.507790 4303 ts_tablet_manager.cc:1397] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:55:47.512514 4301 raft_consensus.cc:348] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.512858 4301 raft_consensus.cc:374] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:47.512977 4301 raft_consensus.cc:729] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671e9de0011c464e8b40942eec847f5a, State: Initialized, Role: FOLLOWER
I20260808 15:55:47.513592 4301 consensus_queue.cc:260] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.515028 4301 ts_tablet_manager.cc:1428] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260808 15:55:47.515525 4302 raft_consensus.cc:348] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.515913 4302 raft_consensus.cc:374] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:47.516021 4302 raft_consensus.cc:729] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Initialized, Role: FOLLOWER
I20260808 15:55:47.516647 4302 consensus_queue.cc:260] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.518314 4302 ts_tablet_manager.cc:1428] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260808 15:55:47.518550 4303 raft_consensus.cc:348] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.518947 4303 raft_consensus.cc:374] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:47.519044 4303 raft_consensus.cc:729] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a56e55d31744ee7955c1c1b124a8a35, State: Initialized, Role: FOLLOWER
I20260808 15:55:47.519650 4303 consensus_queue.cc:260] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.521236 4303 ts_tablet_manager.cc:1428] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260808 15:55:47.526968 4277 tablet.cc:2369] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:47.682670 4308 raft_consensus.cc:482] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:47.683025 4308 raft_consensus.cc:504] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.684790 4308 leader_election.cc:288] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329), 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733)
I20260808 15:55:47.692029 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.685228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:60363 (local address 127.27.254.2:35329)
0808 15:55:47.685463 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.685469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.686432 (+ 963us) server_negotiation.cc:407] Connection header received
0808 15:55:47.686698 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.686702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.686782 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.686898 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:47.687470 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.688166 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.690174 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.691055 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.691254 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.691334 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.691664 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.691786 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:55:47.692042 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.685307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:59077 (local address 127.27.254.1:46733)
0808 15:55:47.685601 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.685610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.686467 (+ 857us) server_negotiation.cc:407] Connection header received
0808 15:55:47.686699 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.686704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.686787 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.686905 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:47.687583 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.688288 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.690152 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.691023 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.691202 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.691286 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.691653 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.691781 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:55:47.692205 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.685315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:46733 (local address 127.27.254.3:59077)
0808 15:55:47.686356 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:47.686371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:47.686545 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:47.686917 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:47.686922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:47.686964 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:47.687425 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.687433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.688433 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.688439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.689986 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.690013 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.691266 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.691272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.691468 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.691523 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 15:55:47.691625 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:55:47.691755 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":80,"spinlock_wait_cycles":1792,"thread_start_us":758,"threads_started":1}
I20260808 15:55:47.692266 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.685134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35329 (local address 127.27.254.3:60363)
0808 15:55:47.686301 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:47.686326 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:47.686547 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:47.686913 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:47.686918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:47.686947 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:47.687325 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.687341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.688292 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.688300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.690027 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:47.690038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.691305 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.691313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:47.691514 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.691537 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:55:47.691638 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:55:47.691777 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":70,"spinlock_wait_cycles":75008,"thread_start_us":583,"threads_started":1}
I20260808 15:55:47.693241 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c39f952f54c39bf12409a8955d4ff" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a" is_pre_election: true
I20260808 15:55:47.693228 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c39f952f54c39bf12409a8955d4ff" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35" is_pre_election: true
I20260808 15:55:47.693778 3946 raft_consensus.cc:2454] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:47.693775 4087 raft_consensus.cc:2454] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:47.694386 4157 leader_election.cc:302] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a, 9a56e55d31744ee7955c1c1b124a8a35; no voters:
I20260808 15:55:47.694805 4308 raft_consensus.cc:2781] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:47.694964 4308 raft_consensus.cc:482] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:47.695083 4308 raft_consensus.cc:3037] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:47.697485 4308 raft_consensus.cc:504] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.698113 4308 leader_election.cc:288] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Requested vote from peers 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329), 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733)
I20260808 15:55:47.698520 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c39f952f54c39bf12409a8955d4ff" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a"
I20260808 15:55:47.698580 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321c39f952f54c39bf12409a8955d4ff" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35"
I20260808 15:55:47.698696 4087 raft_consensus.cc:3037] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:47.698830 3946 raft_consensus.cc:3037] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:47.700853 4087 raft_consensus.cc:2454] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:47.700981 3946 raft_consensus.cc:2454] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:47.701236 4157 leader_election.cc:302] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:47.701537 4308 raft_consensus.cc:2781] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:47.701923 4308 raft_consensus.cc:686] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Becoming Leader. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Running, Role: LEADER
I20260808 15:55:47.702409 4308 consensus_queue.cc:237] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.706758 3787 catalog_manager.cc:5495] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3). New cstate: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } health_report { overall_health: HEALTHY } } }
W20260808 15:55:47.744665 3999 tablet.cc:2369] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:55:47.755375 4138 tablet.cc:2369] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:55:47.776818 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:47.771529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.27.254.3:40621)
0808 15:55:47.771761 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:47.771765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:47.771784 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:55:47.771973 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:47.771977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:47.772103 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:47.772225 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:47.772653 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.773476 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.774742 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:47.775157 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:47.775301 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:47.775732 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:47.776243 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:47.776394 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:47.776471 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 15:55:47.776549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:55:47.784514 3946 raft_consensus.cc:1261] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Refusing update from remote peer 19f5277bd5934d64852f6b3e4d6ed19e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:47.784534 4087 raft_consensus.cc:1261] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Refusing update from remote peer 19f5277bd5934d64852f6b3e4d6ed19e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:47.785538 4308 consensus_queue.cc:1035] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:47.785827 4312 consensus_queue.cc:1035] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:47.802690 4316 mvcc.cc:205] Tried to move back new op lower bound from 7316293827716775936 to 7316293827392765952. Current Snapshot: MvccSnapshot[applied={T|T < 7316293827716775936}]
I20260808 15:55:47.812940 3787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:47.813210 3787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:47.816893 3787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Sending DeleteTablet for 3 replicas of tablet 321c39f952f54c39bf12409a8955d4ff
I20260808 15:55:47.818267 3926 tablet_service.cc:1514] Processing DeleteTablet for tablet 321c39f952f54c39bf12409a8955d4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:47 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33220
I20260808 15:55:47.818267 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet 321c39f952f54c39bf12409a8955d4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:47 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260808 15:55:47.818993 4206 tablet_service.cc:1514] Processing DeleteTablet for tablet 321c39f952f54c39bf12409a8955d4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:47 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37186
I20260808 15:55:47.820053 3788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:47.820719 3788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:47.828224 3925 tablet_service.cc:1467] Processing CreateTablet for tablet 4677c9dbd7a3472a82aa5d294c07ba2e (DEFAULT_TABLE table=test-table [id=e35e58a4fdc94508b01f8a9e7ca9cce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:47.828806 3925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4677c9dbd7a3472a82aa5d294c07ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:47.828742 4204 tablet_service.cc:1467] Processing CreateTablet for tablet 4677c9dbd7a3472a82aa5d294c07ba2e (DEFAULT_TABLE table=test-table [id=e35e58a4fdc94508b01f8a9e7ca9cce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:47.828903 4066 tablet_service.cc:1467] Processing CreateTablet for tablet 4677c9dbd7a3472a82aa5d294c07ba2e (DEFAULT_TABLE table=test-table [id=e35e58a4fdc94508b01f8a9e7ca9cce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:47.829380 4204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4677c9dbd7a3472a82aa5d294c07ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:47.829478 4066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4677c9dbd7a3472a82aa5d294c07ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:47.837014 4301 tablet_bootstrap.cc:492] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Bootstrap starting.
I20260808 15:55:47.838593 4301 tablet_bootstrap.cc:654] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:47.839656 4303 tablet_bootstrap.cc:492] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Bootstrap starting.
I20260808 15:55:47.841114 4303 tablet_bootstrap.cc:654] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:47.843200 4326 tablet_replica.cc:330] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: stopping tablet replica
I20260808 15:55:47.843469 4301 tablet_bootstrap.cc:492] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: No bootstrap required, opened a new log
I20260808 15:55:47.843609 4301 ts_tablet_manager.cc:1397] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260808 15:55:47.843866 4326 raft_consensus.cc:2229] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:47.844475 4301 raft_consensus.cc:348] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.844724 4301 raft_consensus.cc:374] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:47.844631 4326 raft_consensus.cc:2258] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:47.844822 4301 raft_consensus.cc:729] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671e9de0011c464e8b40942eec847f5a, State: Initialized, Role: FOLLOWER
I20260808 15:55:47.845116 4301 consensus_queue.cc:260] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.848176 4301 ts_tablet_manager.cc:1428] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:55:47.851517 4302 tablet_bootstrap.cc:492] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Bootstrap starting.
I20260808 15:55:47.851519 4326 ts_tablet_manager.cc:1905] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:47.853792 4302 tablet_bootstrap.cc:654] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:47.860770 4326 ts_tablet_manager.cc:1918] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:47.861040 4326 log.cc:1196] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/321c39f952f54c39bf12409a8955d4ff
I20260808 15:55:47.861717 4326 ts_tablet_manager.cc:1939] T 321c39f952f54c39bf12409a8955d4ff P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting consensus metadata
I20260808 15:55:47.861856 4303 tablet_bootstrap.cc:492] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: No bootstrap required, opened a new log
I20260808 15:55:47.862026 4303 ts_tablet_manager.cc:1397] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.007s
I20260808 15:55:47.862926 4303 raft_consensus.cc:348] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.863168 4303 raft_consensus.cc:374] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:47.863281 4303 raft_consensus.cc:729] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a56e55d31744ee7955c1c1b124a8a35, State: Initialized, Role: FOLLOWER
I20260808 15:55:47.863636 4303 consensus_queue.cc:260] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.863833 3772 catalog_manager.cc:4841] TS 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621): tablet 321c39f952f54c39bf12409a8955d4ff (table test-table [id=9d76b778043c47aaba5e5b158b0aa4f7]) successfully deleted
I20260808 15:55:47.864977 4303 ts_tablet_manager.cc:1428] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:55:47.865708 4325 tablet_replica.cc:330] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: stopping tablet replica
I20260808 15:55:47.866178 4325 raft_consensus.cc:2229] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:47.866672 4325 raft_consensus.cc:2258] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:47.867506 4324 tablet_replica.cc:330] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: stopping tablet replica
I20260808 15:55:47.867919 4324 raft_consensus.cc:2229] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:47.868366 4324 raft_consensus.cc:2258] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:47.870116 4302 tablet_bootstrap.cc:492] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: No bootstrap required, opened a new log
I20260808 15:55:47.870278 4302 ts_tablet_manager.cc:1397] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260808 15:55:47.870967 4325 ts_tablet_manager.cc:1905] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:47.871114 4302 raft_consensus.cc:348] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.871348 4302 raft_consensus.cc:374] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:47.871452 4302 raft_consensus.cc:729] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Initialized, Role: FOLLOWER
I20260808 15:55:47.871724 4302 consensus_queue.cc:260] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:47.872393 4302 ts_tablet_manager.cc:1428] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:55:47.872874 4324 ts_tablet_manager.cc:1905] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:47.878299 4324 ts_tablet_manager.cc:1918] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:47.878414 4324 log.cc:1196] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/321c39f952f54c39bf12409a8955d4ff
I20260808 15:55:47.878867 4324 ts_tablet_manager.cc:1939] T 321c39f952f54c39bf12409a8955d4ff P 9a56e55d31744ee7955c1c1b124a8a35: Deleting consensus metadata
I20260808 15:55:47.880934 3774 catalog_manager.cc:4841] TS 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733): tablet 321c39f952f54c39bf12409a8955d4ff (table test-table [id=9d76b778043c47aaba5e5b158b0aa4f7]) successfully deleted
I20260808 15:55:47.884734 4325 ts_tablet_manager.cc:1918] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:47.884892 4325 log.cc:1196] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/321c39f952f54c39bf12409a8955d4ff
I20260808 15:55:47.885426 4325 ts_tablet_manager.cc:1939] T 321c39f952f54c39bf12409a8955d4ff P 671e9de0011c464e8b40942eec847f5a: Deleting consensus metadata
I20260808 15:55:47.889554 3772 catalog_manager.cc:4841] TS 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329): tablet 321c39f952f54c39bf12409a8955d4ff (table test-table [id=9d76b778043c47aaba5e5b158b0aa4f7]) successfully deleted
I20260808 15:55:48.073886 4312 raft_consensus.cc:482] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:48.074177 4312 raft_consensus.cc:504] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.075007 4312 leader_election.cc:288] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:48.075670 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4677c9dbd7a3472a82aa5d294c07ba2e" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a" is_pre_election: true
I20260808 15:55:48.075700 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4677c9dbd7a3472a82aa5d294c07ba2e" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35" is_pre_election: true
I20260808 15:55:48.075933 4087 raft_consensus.cc:2454] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:48.075995 3946 raft_consensus.cc:2454] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:48.076453 4157 leader_election.cc:302] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:48.076759 4312 raft_consensus.cc:2781] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:48.076872 4312 raft_consensus.cc:482] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:48.076975 4312 raft_consensus.cc:3037] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.080458 4312 raft_consensus.cc:504] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.081159 4312 leader_election.cc:288] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Requested vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:48.081736 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4677c9dbd7a3472a82aa5d294c07ba2e" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a"
I20260808 15:55:48.081736 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4677c9dbd7a3472a82aa5d294c07ba2e" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35"
I20260808 15:55:48.081949 3946 raft_consensus.cc:3037] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.081949 4087 raft_consensus.cc:3037] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.084897 3946 raft_consensus.cc:2454] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:48.084897 4087 raft_consensus.cc:2454] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:48.085345 4157 leader_election.cc:302] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:48.085705 4312 raft_consensus.cc:2781] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:48.085884 4312 raft_consensus.cc:686] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Becoming Leader. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Running, Role: LEADER
I20260808 15:55:48.086134 4312 consensus_queue.cc:237] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.089211 3788 catalog_manager.cc:5495] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3). New cstate: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } health_report { overall_health: HEALTHY } } }
Aug 08 15:55:48 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204544, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.62@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:48.187040 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.171742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33183 (local address 127.0.0.1:45360)
0808 15:55:48.172976 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:48.172992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:48.173126 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:48.173563 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:48.173572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:48.174208 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:48.174380 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.174388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.175962 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.175971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.176906 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.176916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.177924 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.177928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.178107 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.178754 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:48.178771 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:48.181303 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:48.184470 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:48.184482 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:48.184488 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:48.185046 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:48.185383 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:48.185390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:48.185394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:48.185619 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:48.186282 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:48.186288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:48.186427 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 15:55:48.186562 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 15:55:48.186754 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":31,"thread_start_us":897,"threads_started":1}
I20260808 15:55:48.187393 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.171833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.27.254.62:33183)
0808 15:55:48.172907 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.172913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.173096 (+ 183us) server_negotiation.cc:407] Connection header received
0808 15:55:48.173240 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.173246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.173325 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.173485 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:48.174540 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.175821 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.177052 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.177767 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.177907 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.181514 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:48.181519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:48.181522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:48.181557 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:48.184336 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:48.185150 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:48.185153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:48.185155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:48.185255 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:48.185768 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:48.185773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:48.185776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:48.186137 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:48.186280 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:48.186715 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 15:55:48.186934 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":845,"thread_start_us":625,"threads_started":1}
I20260808 15:55:48.192202 3788 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:45360:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:55:48.193325 3788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:48.200266 3926 tablet_service.cc:1467] Processing CreateTablet for tablet 323a7325d04441d5ac4b72162dc27caf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726ff5f4e72e4c21b277f53d08843dd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:55:48.200817 3926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323a7325d04441d5ac4b72162dc27caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:48.201047 4204 tablet_service.cc:1467] Processing CreateTablet for tablet 323a7325d04441d5ac4b72162dc27caf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726ff5f4e72e4c21b277f53d08843dd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:55:48.201592 4204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323a7325d04441d5ac4b72162dc27caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:48.206494 4303 tablet_bootstrap.cc:492] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: Bootstrap starting.
I20260808 15:55:48.207991 4303 tablet_bootstrap.cc:654] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:48.208724 4067 tablet_service.cc:1467] Processing CreateTablet for tablet 323a7325d04441d5ac4b72162dc27caf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726ff5f4e72e4c21b277f53d08843dd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:55:48.209218 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323a7325d04441d5ac4b72162dc27caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:48.210210 4303 tablet_bootstrap.cc:492] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: No bootstrap required, opened a new log
I20260808 15:55:48.210383 4303 ts_tablet_manager.cc:1397] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260808 15:55:48.210942 4303 raft_consensus.cc:348] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.211155 4303 raft_consensus.cc:374] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:48.211254 4303 raft_consensus.cc:729] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a56e55d31744ee7955c1c1b124a8a35, State: Initialized, Role: FOLLOWER
I20260808 15:55:48.211494 4303 consensus_queue.cc:260] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.211925 4309 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.212247 4309 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.213666 4303 ts_tablet_manager.cc:1428] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 15:55:48.216264 4301 tablet_bootstrap.cc:492] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: Bootstrap starting.
I20260808 15:55:48.216338 4302 tablet_bootstrap.cc:492] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: Bootstrap starting.
I20260808 15:55:48.217971 4302 tablet_bootstrap.cc:654] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:48.218021 4301 tablet_bootstrap.cc:654] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:48.220253 4301 tablet_bootstrap.cc:492] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: No bootstrap required, opened a new log
I20260808 15:55:48.220423 4301 ts_tablet_manager.cc:1397] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 15:55:48.220432 4302 tablet_bootstrap.cc:492] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: No bootstrap required, opened a new log
I20260808 15:55:48.220592 4302 ts_tablet_manager.cc:1397] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:55:48.221064 4301 raft_consensus.cc:348] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.221266 4301 raft_consensus.cc:374] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:48.221364 4301 raft_consensus.cc:729] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671e9de0011c464e8b40942eec847f5a, State: Initialized, Role: FOLLOWER
I20260808 15:55:48.221352 4302 raft_consensus.cc:348] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.221750 4302 raft_consensus.cc:374] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:48.221621 4301 consensus_queue.cc:260] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.221865 4302 raft_consensus.cc:729] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Initialized, Role: FOLLOWER
I20260808 15:55:48.222141 4301 ts_tablet_manager.cc:1428] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 15:55:48.222092 4302 consensus_queue.cc:260] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.222470 4307 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.222563 4312 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.222798 4307 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.222839 4312 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.223338 4302 ts_tablet_manager.cc:1428] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:48.484663 4312 consensus_queue.cc:1035] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:48.489878 4308 consensus_queue.cc:1035] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:48.614107 4308 raft_consensus.cc:482] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:48.614393 4308 raft_consensus.cc:504] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.615042 4308 leader_election.cc:288] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:48.615628 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a7325d04441d5ac4b72162dc27caf" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a" is_pre_election: true
I20260808 15:55:48.615643 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a7325d04441d5ac4b72162dc27caf" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35" is_pre_election: true
I20260808 15:55:48.615866 4087 raft_consensus.cc:2454] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:48.615923 3946 raft_consensus.cc:2454] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:48.616364 4157 leader_election.cc:302] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:48.616680 4308 raft_consensus.cc:2781] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:48.616792 4308 raft_consensus.cc:482] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:48.616901 4308 raft_consensus.cc:3037] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.618949 4308 raft_consensus.cc:504] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.619519 4308 leader_election.cc:288] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Requested vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:48.619881 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a7325d04441d5ac4b72162dc27caf" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35"
I20260808 15:55:48.619982 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a7325d04441d5ac4b72162dc27caf" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a"
I20260808 15:55:48.620136 3946 raft_consensus.cc:3037] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.620193 4087 raft_consensus.cc:3037] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.622143 4087 raft_consensus.cc:2454] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:48.622205 3946 raft_consensus.cc:2454] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:48.622527 4157 leader_election.cc:302] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:48.622838 4308 raft_consensus.cc:2781] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:48.622997 4308 raft_consensus.cc:686] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Becoming Leader. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Running, Role: LEADER
I20260808 15:55:48.623302 4308 consensus_queue.cc:237] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } }
I20260808 15:55:48.624787 4344 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: TxnStatusTablet state changed. Reason: New leader 19f5277bd5934d64852f6b3e4d6ed19e. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.625064 4344 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:55:48.625703 4348 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:55:48.625917 4348 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:55:48.626274 3788 catalog_manager.cc:5495] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3). New cstate: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:48.667694 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.660951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.27.254.3:40621)
0808 15:55:48.661802 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.661808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.661831 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:55:48.661915 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.661922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.661999 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.662576 (+ 577us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:48.662786 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.664004 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.665900 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.666350 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.666457 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.666971 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:48.667111 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:48.667180 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:48.667354 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 15:55:48.667412 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":507,"threads_started":1}
I20260808 15:55:48.667810 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.660428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:40621 (local address 127.0.0.1:37202)
0808 15:55:48.660939 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:48.660960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:48.661100 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:48.662192 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:48.662198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:48.662213 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:55:48.662437 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.662451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.664164 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.664170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.665773 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.665790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.666509 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.666516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.666706 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.666787 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:55:48.667219 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:55:48.667240 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 15:55:48.667329 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 15:55:48.667476 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260808 15:55:48.671350 3946 raft_consensus.cc:1261] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Refusing update from remote peer 19f5277bd5934d64852f6b3e4d6ed19e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:48.671360 4087 raft_consensus.cc:1261] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Refusing update from remote peer 19f5277bd5934d64852f6b3e4d6ed19e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:48.671995 4308 consensus_queue.cc:1035] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:48.672291 4344 consensus_queue.cc:1035] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:48.675985 4307 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: TxnStatusTablet state changed. Reason: New leader 19f5277bd5934d64852f6b3e4d6ed19e. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.676254 4307 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.677069 4309 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: TxnStatusTablet state changed. Reason: New leader 19f5277bd5934d64852f6b3e4d6ed19e. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.677317 4309 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.678788 4342 mvcc.cc:205] Tried to move back new op lower bound from 7316293831351607296 to 7316293831162892288. Current Snapshot: MvccSnapshot[applied={T|T < 7316293831351607296}]
I20260808 15:55:48.685920 4344 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.686249 4344 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:55:48.687611 4344 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.687886 4344 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 19f5277bd5934d64852f6b3e4d6ed19e: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:55:48.696514 4307 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.696780 4307 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 671e9de0011c464e8b40942eec847f5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.699390 4309 tablet_replica.cc:439] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } }
I20260808 15:55:48.699680 4309 tablet_replica.cc:442] T 323a7325d04441d5ac4b72162dc27caf P 9a56e55d31744ee7955c1c1b124a8a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:55:48.711396 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.705871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:42691 (local address 127.27.254.3:40621)
0808 15:55:48.706024 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.706028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.706710 (+ 682us) server_negotiation.cc:407] Connection header received
0808 15:55:48.706910 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.706914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.706974 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.707061 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:48.707557 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.708273 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.709698 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.710140 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.710279 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.710686 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:48.710774 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:48.710875 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:48.711007 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 15:55:48.711074 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:55:48.711473 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.705987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:40621 (local address 127.27.254.3:42691)
0808 15:55:48.706603 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:48.706621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:48.706788 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:48.707070 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:48.707075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:48.707084 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:55:48.707411 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.707420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.708397 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.708405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.709567 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.709579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.710297 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.710303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.710459 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.710505 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:55:48.710875 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:55:48.710887 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 15:55:48.710990 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:55:48.711115 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":326,"threads_started":1}
I20260808 15:55:48.774439 3787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:48.774729 3787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:48.777920 3787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Sending DeleteTablet for 3 replicas of tablet 4677c9dbd7a3472a82aa5d294c07ba2e
I20260808 15:55:48.778611 3926 tablet_service.cc:1514] Processing DeleteTablet for tablet 4677c9dbd7a3472a82aa5d294c07ba2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:48 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33220
I20260808 15:55:48.778874 4206 tablet_service.cc:1514] Processing DeleteTablet for tablet 4677c9dbd7a3472a82aa5d294c07ba2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:48 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37186
I20260808 15:55:48.779001 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet 4677c9dbd7a3472a82aa5d294c07ba2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:48 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260808 15:55:48.793696 4362 tablet_replica.cc:330] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: stopping tablet replica
I20260808 15:55:48.794114 4362 raft_consensus.cc:2229] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:48.795298 4362 raft_consensus.cc:2258] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:48.797477 4362 ts_tablet_manager.cc:1905] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:48.805850 4361 tablet_replica.cc:330] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: stopping tablet replica
I20260808 15:55:48.805998 4363 tablet_replica.cc:330] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: stopping tablet replica
I20260808 15:55:48.806278 4363 raft_consensus.cc:2229] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:48.806581 4363 raft_consensus.cc:2258] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:48.807801 4363 ts_tablet_manager.cc:1905] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:48.809614 4361 raft_consensus.cc:2229] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:48.809955 4361 raft_consensus.cc:2258] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:48.811093 4361 ts_tablet_manager.cc:1905] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:48.814798 4363 ts_tablet_manager.cc:1918] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:48.815060 4363 log.cc:1196] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/4677c9dbd7a3472a82aa5d294c07ba2e
I20260808 15:55:48.815642 4363 ts_tablet_manager.cc:1939] T 4677c9dbd7a3472a82aa5d294c07ba2e P 671e9de0011c464e8b40942eec847f5a: Deleting consensus metadata
I20260808 15:55:48.816926 3772 catalog_manager.cc:4841] TS 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329): tablet 4677c9dbd7a3472a82aa5d294c07ba2e (table test-table [id=e35e58a4fdc94508b01f8a9e7ca9cce0]) successfully deleted
I20260808 15:55:48.817929 4361 ts_tablet_manager.cc:1918] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:48.818153 4361 log.cc:1196] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/4677c9dbd7a3472a82aa5d294c07ba2e
I20260808 15:55:48.818732 4361 ts_tablet_manager.cc:1939] T 4677c9dbd7a3472a82aa5d294c07ba2e P 9a56e55d31744ee7955c1c1b124a8a35: Deleting consensus metadata
I20260808 15:55:48.819798 3774 catalog_manager.cc:4841] TS 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733): tablet 4677c9dbd7a3472a82aa5d294c07ba2e (table test-table [id=e35e58a4fdc94508b01f8a9e7ca9cce0]) successfully deleted
I20260808 15:55:48.821188 4362 ts_tablet_manager.cc:1918] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:48.821334 4362 log.cc:1196] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/4677c9dbd7a3472a82aa5d294c07ba2e
I20260808 15:55:48.821887 4362 ts_tablet_manager.cc:1939] T 4677c9dbd7a3472a82aa5d294c07ba2e P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting consensus metadata
I20260808 15:55:48.822965 3772 catalog_manager.cc:4841] TS 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621): tablet 4677c9dbd7a3472a82aa5d294c07ba2e (table test-table [id=e35e58a4fdc94508b01f8a9e7ca9cce0]) successfully deleted
Aug 08 15:55:48 dist-test-slave-srgv krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:55:48 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:48 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:55:48.853152 4371 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:48.841487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.27.254.62:33183)
0808 15:55:48.842330 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.842338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.842360 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:55:48.842428 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.842456 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.842537 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.842652 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:48.843944 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.845174 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.846455 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.847120 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.847250 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.852897 (+ 5647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45366: BlockingRecv error: recv got EOF from 127.0.0.1:45366 (error 108)
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":549,"threads_started":1}
Aug 08 15:55:48 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM
I20260808 15:55:48.869735 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.855968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.27.254.62:33183)
0808 15:55:48.856196 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.856206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.856670 (+ 464us) server_negotiation.cc:407] Connection header received
0808 15:55:48.856865 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.856870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.856938 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.857036 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:48.858259 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.858978 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.860222 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.860653 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.860787 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.864602 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:48.864607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:48.864610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:48.864642 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:48.867147 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:48.867816 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:48.867822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:48.867826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:48.867963 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:48.868323 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:48.868328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:48.868331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:48.868606 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:48.868880 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:48.869180 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 15:55:48.869390 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:55:48.873535 3787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:48.874251 3787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:48.881305 3926 tablet_service.cc:1467] Processing CreateTablet for tablet 62abe578bbdb4f8594d8b675a71ddbe6 (DEFAULT_TABLE table=test-table [id=b192ba057b82442da73b5a7707f25870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:48.881956 3926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62abe578bbdb4f8594d8b675a71ddbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:48.882089 4206 tablet_service.cc:1467] Processing CreateTablet for tablet 62abe578bbdb4f8594d8b675a71ddbe6 (DEFAULT_TABLE table=test-table [id=b192ba057b82442da73b5a7707f25870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:48.882352 4067 tablet_service.cc:1467] Processing CreateTablet for tablet 62abe578bbdb4f8594d8b675a71ddbe6 (DEFAULT_TABLE table=test-table [id=b192ba057b82442da73b5a7707f25870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:48.882527 4206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62abe578bbdb4f8594d8b675a71ddbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:48.882750 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62abe578bbdb4f8594d8b675a71ddbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:48.895246 4377 tablet_bootstrap.cc:492] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Bootstrap starting.
I20260808 15:55:48.898972 4378 tablet_bootstrap.cc:492] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Bootstrap starting.
I20260808 15:55:48.899485 4377 tablet_bootstrap.cc:654] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:48.900019 4379 tablet_bootstrap.cc:492] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Bootstrap starting.
I20260808 15:55:48.900864 4378 tablet_bootstrap.cc:654] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:48.902240 4379 tablet_bootstrap.cc:654] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:48.906510 4378 tablet_bootstrap.cc:492] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: No bootstrap required, opened a new log
I20260808 15:55:48.906711 4378 ts_tablet_manager.cc:1397] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:55:48.907842 4378 raft_consensus.cc:348] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.908325 4378 raft_consensus.cc:374] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:48.908562 4378 raft_consensus.cc:729] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671e9de0011c464e8b40942eec847f5a, State: Initialized, Role: FOLLOWER
I20260808 15:55:48.908937 4378 consensus_queue.cc:260] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.910233 4379 tablet_bootstrap.cc:492] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: No bootstrap required, opened a new log
I20260808 15:55:48.910415 4379 ts_tablet_manager.cc:1397] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260808 15:55:48.910926 4377 tablet_bootstrap.cc:492] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: No bootstrap required, opened a new log
I20260808 15:55:48.911132 4377 ts_tablet_manager.cc:1397] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.008s
I20260808 15:55:48.911139 4379 raft_consensus.cc:348] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.911443 4379 raft_consensus.cc:374] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:48.911576 4379 raft_consensus.cc:729] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a56e55d31744ee7955c1c1b124a8a35, State: Initialized, Role: FOLLOWER
I20260808 15:55:48.911847 4379 consensus_queue.cc:260] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.912109 4378 ts_tablet_manager.cc:1428] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 15:55:48.913053 4379 ts_tablet_manager.cc:1428] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 15:55:48.913072 4377 raft_consensus.cc:348] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.913360 4377 raft_consensus.cc:374] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:48.913570 4377 raft_consensus.cc:729] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Initialized, Role: FOLLOWER
I20260808 15:55:48.913921 4377 consensus_queue.cc:260] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.915403 4377 ts_tablet_manager.cc:1428] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260808 15:55:48.954877 4307 raft_consensus.cc:482] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:48.955181 4307 raft_consensus.cc:504] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.957304 4307 leader_election.cc:288] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621)
I20260808 15:55:48.988124 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.957731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:46733 (local address 127.27.254.2:37901)
0808 15:55:48.974492 (+ 16761us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:48.974522 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:48.974695 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:48.978640 (+ 3945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:48.978647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:48.978685 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:48.979131 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.979146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.981058 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.981067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.983808 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.983831 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.987004 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.987012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.987244 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.987296 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 15:55:48.987405 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:55:48.987597 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16590,"thread_start_us":26398,"threads_started":1}
I20260808 15:55:48.989099 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.958260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:37901 (local address 127.27.254.1:46733)
0808 15:55:48.977980 (+ 19720us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.977990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.978017 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:55:48.978099 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.978107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.978198 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.978315 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:48.979379 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.980840 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.984884 (+ 4044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.986504 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.986750 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.986859 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:48.988391 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0808 15:55:48.988565 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19542,"thread_start_us":19376,"threads_started":1}
I20260808 15:55:48.990223 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62abe578bbdb4f8594d8b675a71ddbe6" candidate_uuid: "671e9de0011c464e8b40942eec847f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35" is_pre_election: true
I20260808 15:55:48.990520 3946 raft_consensus.cc:2454] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671e9de0011c464e8b40942eec847f5a in term 0.
I20260808 15:55:48.991277 4020 leader_election.cc:302] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671e9de0011c464e8b40942eec847f5a, 9a56e55d31744ee7955c1c1b124a8a35; no voters:
I20260808 15:55:48.991806 4307 raft_consensus.cc:2781] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:48.991940 4307 raft_consensus.cc:482] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:48.992048 4307 raft_consensus.cc:3037] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.992342 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.957679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:36981 (local address 127.27.254.3:40621)
0808 15:55:48.957870 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:48.957877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:48.984653 (+ 26776us) server_negotiation.cc:407] Connection header received
0808 15:55:48.984853 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:48.984859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:48.984965 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:48.985066 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:55:48.985939 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.987109 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.989660 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.991012 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.991184 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.991257 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:48.991895 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 15:55:48.992049 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:55:48.993085 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:48.957758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:40621 (local address 127.27.254.2:36981)
0808 15:55:48.984509 (+ 26751us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:48.984534 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:48.984705 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:48.985252 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:48.985257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:48.985286 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:55:48.985701 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.985715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.987323 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.987329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.989478 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:48.989495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:48.991487 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:48.991494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:48.991686 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:48.991724 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:55:48.992579 (+ 855us) client_negotiation.cc:240] Negotiation successful
0808 15:55:48.992728 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26589,"thread_start_us":17031,"threads_started":1}
I20260808 15:55:48.993906 4231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62abe578bbdb4f8594d8b675a71ddbe6" candidate_uuid: "671e9de0011c464e8b40942eec847f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" is_pre_election: true
I20260808 15:55:48.994273 4231 raft_consensus.cc:2454] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671e9de0011c464e8b40942eec847f5a in term 0.
I20260808 15:55:48.995862 4307 raft_consensus.cc:504] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:48.996620 4307 leader_election.cc:288] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [CANDIDATE]: Term 1 election: Requested vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621)
I20260808 15:55:48.997241 4231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62abe578bbdb4f8594d8b675a71ddbe6" candidate_uuid: "671e9de0011c464e8b40942eec847f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f5277bd5934d64852f6b3e4d6ed19e"
I20260808 15:55:48.997473 4231 raft_consensus.cc:3037] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:48.997902 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62abe578bbdb4f8594d8b675a71ddbe6" candidate_uuid: "671e9de0011c464e8b40942eec847f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35"
I20260808 15:55:48.998171 3946 raft_consensus.cc:3037] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:49.000911 4231 raft_consensus.cc:2454] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671e9de0011c464e8b40942eec847f5a in term 1.
I20260808 15:55:49.001609 4018 leader_election.cc:302] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:49.001844 3946 raft_consensus.cc:2454] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671e9de0011c464e8b40942eec847f5a in term 1.
I20260808 15:55:49.002249 4307 raft_consensus.cc:2781] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:49.002821 4307 raft_consensus.cc:686] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 1 LEADER]: Becoming Leader. State: Replica: 671e9de0011c464e8b40942eec847f5a, State: Running, Role: LEADER
I20260808 15:55:49.003417 4307 consensus_queue.cc:237] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.007982 3788 catalog_manager.cc:5495] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 671e9de0011c464e8b40942eec847f5a (127.27.254.2). New cstate: current_term: 1 leader_uuid: "671e9de0011c464e8b40942eec847f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } health_report { overall_health: UNKNOWN } } }
I20260808 15:55:49.031140 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:49.022102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.27.254.2:35329)
0808 15:55:49.023202 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:49.023207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:49.023231 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:55:49.023327 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:49.023332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:49.023435 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:49.023621 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:49.024127 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.025278 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.027682 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.028328 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.028462 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:49.029307 (+ 845us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:49.030237 (+ 930us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:49.030403 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:49.030555 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 15:55:49.030658 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":800,"threads_started":1}
I20260808 15:55:49.035576 4231 raft_consensus.cc:1261] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Refusing update from remote peer 671e9de0011c464e8b40942eec847f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:49.036476 3946 raft_consensus.cc:1261] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Refusing update from remote peer 671e9de0011c464e8b40942eec847f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:49.036564 4386 consensus_queue.cc:1035] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:49.037711 4386 consensus_queue.cc:1035] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:49.054786 4346 mvcc.cc:205] Tried to move back new op lower bound from 7316293832841011200 to 7316293832722952192. Current Snapshot: MvccSnapshot[applied={T|T < 7316293832841011200}]
I20260808 15:55:49.077059 3788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:49.077376 3788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:49.081811 3788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Sending DeleteTablet for 3 replicas of tablet 62abe578bbdb4f8594d8b675a71ddbe6
I20260808 15:55:49.082674 3926 tablet_service.cc:1514] Processing DeleteTablet for tablet 62abe578bbdb4f8594d8b675a71ddbe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:49 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33220
I20260808 15:55:49.082968 4361 tablet_replica.cc:330] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: stopping tablet replica
I20260808 15:55:49.083015 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet 62abe578bbdb4f8594d8b675a71ddbe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:49 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260808 15:55:49.083257 4361 raft_consensus.cc:2229] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:49.083328 4363 tablet_replica.cc:330] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: stopping tablet replica
I20260808 15:55:49.083482 4361 raft_consensus.cc:2258] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:49.083510 4363 raft_consensus.cc:2229] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:49.084014 4363 raft_consensus.cc:2258] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:49.084830 4361 ts_tablet_manager.cc:1905] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:49.085144 4363 ts_tablet_manager.cc:1905] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:49.085594 4206 tablet_service.cc:1514] Processing DeleteTablet for tablet 62abe578bbdb4f8594d8b675a71ddbe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:49 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37186
I20260808 15:55:49.086313 3787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:55:49.086936 3787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:55:49.090135 4362 tablet_replica.cc:330] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: stopping tablet replica
I20260808 15:55:49.090348 4362 raft_consensus.cc:2229] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:49.090593 4362 raft_consensus.cc:2258] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:49.091589 4362 ts_tablet_manager.cc:1905] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:49.091974 4361 ts_tablet_manager.cc:1918] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:49.092139 4361 log.cc:1196] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/62abe578bbdb4f8594d8b675a71ddbe6
I20260808 15:55:49.092556 4361 ts_tablet_manager.cc:1939] T 62abe578bbdb4f8594d8b675a71ddbe6 P 9a56e55d31744ee7955c1c1b124a8a35: Deleting consensus metadata
I20260808 15:55:49.093511 4363 ts_tablet_manager.cc:1918] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:49.093684 3774 catalog_manager.cc:4841] TS 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733): tablet 62abe578bbdb4f8594d8b675a71ddbe6 (table test-table [id=b192ba057b82442da73b5a7707f25870]) successfully deleted
I20260808 15:55:49.093739 4363 log.cc:1196] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/62abe578bbdb4f8594d8b675a71ddbe6
I20260808 15:55:49.094244 4363 ts_tablet_manager.cc:1939] T 62abe578bbdb4f8594d8b675a71ddbe6 P 671e9de0011c464e8b40942eec847f5a: Deleting consensus metadata
I20260808 15:55:49.096139 4206 tablet_service.cc:1467] Processing CreateTablet for tablet d6aadb7742ba4cb8835b9d64d76cf3be (DEFAULT_TABLE table=test-table [id=96b1a21c95fb4305a946c1c8b1a70d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:49.096139 4067 tablet_service.cc:1467] Processing CreateTablet for tablet d6aadb7742ba4cb8835b9d64d76cf3be (DEFAULT_TABLE table=test-table [id=96b1a21c95fb4305a946c1c8b1a70d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:49.096601 4206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6aadb7742ba4cb8835b9d64d76cf3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:49.096661 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6aadb7742ba4cb8835b9d64d76cf3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:49.098667 3772 catalog_manager.cc:4841] TS 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329): tablet 62abe578bbdb4f8594d8b675a71ddbe6 (table test-table [id=b192ba057b82442da73b5a7707f25870]) successfully deleted
I20260808 15:55:49.098740 4362 ts_tablet_manager.cc:1918] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:55:49.098875 4362 log.cc:1196] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/62abe578bbdb4f8594d8b675a71ddbe6
I20260808 15:55:49.099303 4362 ts_tablet_manager.cc:1939] T 62abe578bbdb4f8594d8b675a71ddbe6 P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting consensus metadata
I20260808 15:55:49.100268 3772 catalog_manager.cc:4841] TS 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621): tablet 62abe578bbdb4f8594d8b675a71ddbe6 (table test-table [id=b192ba057b82442da73b5a7707f25870]) successfully deleted
I20260808 15:55:49.101977 3926 tablet_service.cc:1467] Processing CreateTablet for tablet d6aadb7742ba4cb8835b9d64d76cf3be (DEFAULT_TABLE table=test-table [id=96b1a21c95fb4305a946c1c8b1a70d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:55:49.102403 3926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6aadb7742ba4cb8835b9d64d76cf3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:49.104521 4378 tablet_bootstrap.cc:492] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Bootstrap starting.
I20260808 15:55:49.106307 4378 tablet_bootstrap.cc:654] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:49.108508 4379 tablet_bootstrap.cc:492] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Bootstrap starting.
I20260808 15:55:49.110146 4379 tablet_bootstrap.cc:654] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:49.111107 4377 tablet_bootstrap.cc:492] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Bootstrap starting.
I20260808 15:55:49.113276 4377 tablet_bootstrap.cc:654] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:49.125535 4379 tablet_bootstrap.cc:492] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: No bootstrap required, opened a new log
I20260808 15:55:49.125743 4379 ts_tablet_manager.cc:1397] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20260808 15:55:49.126037 4377 tablet_bootstrap.cc:492] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: No bootstrap required, opened a new log
I20260808 15:55:49.126238 4377 ts_tablet_manager.cc:1397] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260808 15:55:49.126721 4379 raft_consensus.cc:348] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.127045 4379 raft_consensus.cc:374] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:49.127149 4379 raft_consensus.cc:729] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a56e55d31744ee7955c1c1b124a8a35, State: Initialized, Role: FOLLOWER
I20260808 15:55:49.127200 4377 raft_consensus.cc:348] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.127498 4377 raft_consensus.cc:374] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:49.127586 4377 raft_consensus.cc:729] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Initialized, Role: FOLLOWER
I20260808 15:55:49.127588 4379 consensus_queue.cc:260] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.127852 4377 consensus_queue.cc:260] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.129694 4377 ts_tablet_manager.cc:1428] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:49.131310 4379 ts_tablet_manager.cc:1428] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 15:55:49.133446 4378 tablet_bootstrap.cc:492] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: No bootstrap required, opened a new log
I20260808 15:55:49.133677 4378 ts_tablet_manager.cc:1397] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20260808 15:55:49.134492 4378 raft_consensus.cc:348] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.134763 4378 raft_consensus.cc:374] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:49.134837 4378 raft_consensus.cc:729] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671e9de0011c464e8b40942eec847f5a, State: Initialized, Role: FOLLOWER
I20260808 15:55:49.135098 4378 consensus_queue.cc:260] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.136183 4378 ts_tablet_manager.cc:1428] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:55:49.146157 4344 raft_consensus.cc:482] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:55:49.146384 4344 raft_consensus.cc:504] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.147168 4344 leader_election.cc:288] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:49.147920 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6aadb7742ba4cb8835b9d64d76cf3be" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a" is_pre_election: true
I20260808 15:55:49.148250 4087 raft_consensus.cc:2454] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:49.149168 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6aadb7742ba4cb8835b9d64d76cf3be" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35" is_pre_election: true
I20260808 15:55:49.149485 3946 raft_consensus.cc:2454] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 0.
I20260808 15:55:49.150225 4157 leader_election.cc:302] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 671e9de0011c464e8b40942eec847f5a; no voters:
I20260808 15:55:49.151708 4344 raft_consensus.cc:2781] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:55:49.151824 4344 raft_consensus.cc:482] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:55:49.151957 4344 raft_consensus.cc:3037] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:49.156896 4344 raft_consensus.cc:504] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.158226 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6aadb7742ba4cb8835b9d64d76cf3be" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a56e55d31744ee7955c1c1b124a8a35"
I20260808 15:55:49.158500 3946 raft_consensus.cc:3037] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:49.162384 3946 raft_consensus.cc:2454] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:49.162943 4159 leader_election.cc:302] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f5277bd5934d64852f6b3e4d6ed19e, 9a56e55d31744ee7955c1c1b124a8a35; no voters:
I20260808 15:55:49.163276 4312 raft_consensus.cc:2781] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:49.163434 4312 raft_consensus.cc:686] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Becoming Leader. State: Replica: 19f5277bd5934d64852f6b3e4d6ed19e, State: Running, Role: LEADER
I20260808 15:55:49.163669 4312 consensus_queue.cc:237] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } }
I20260808 15:55:49.163949 4087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6aadb7742ba4cb8835b9d64d76cf3be" candidate_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671e9de0011c464e8b40942eec847f5a"
I20260808 15:55:49.164227 4087 raft_consensus.cc:3037] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:49.167981 4087 raft_consensus.cc:2454] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f5277bd5934d64852f6b3e4d6ed19e in term 1.
I20260808 15:55:49.170610 4344 leader_election.cc:288] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [CANDIDATE]: Term 1 election: Requested vote from peers 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733), 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329)
I20260808 15:55:49.172821 3786 catalog_manager.cc:5495] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3). New cstate: current_term: 1 leader_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f5277bd5934d64852f6b3e4d6ed19e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40621 } health_report { overall_health: HEALTHY } } }
I20260808 15:55:49.206055 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:49.199527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.27.254.3:40621)
0808 15:55:49.199695 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:49.199702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:49.199719 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:55:49.199778 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:49.199783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:49.199859 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:49.199942 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:55:49.200504 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.201589 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.203561 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.204192 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.204309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:49.204883 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:55:49.205026 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:55:49.205119 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:49.205670 (+ 551us) server_negotiation.cc:299] Negotiation successful
0808 15:55:49.205749 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:55:49.217175 4086 raft_consensus.cc:1261] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Refusing update from remote peer 19f5277bd5934d64852f6b3e4d6ed19e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:49.217867 3946 raft_consensus.cc:1261] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Refusing update from remote peer 19f5277bd5934d64852f6b3e4d6ed19e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:55:49.218211 4344 consensus_queue.cc:1035] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671e9de0011c464e8b40942eec847f5a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:49.218719 4344 consensus_queue.cc:1035] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a56e55d31744ee7955c1c1b124a8a35" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:55:49.317595 3787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:49.317910 3787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:55:49.321928 3787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c54ac9f031a743d98616435a98a64f64: Sending DeleteTablet for 3 replicas of tablet d6aadb7742ba4cb8835b9d64d76cf3be
I20260808 15:55:49.323081 4067 tablet_service.cc:1514] Processing DeleteTablet for tablet d6aadb7742ba4cb8835b9d64d76cf3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:49 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260808 15:55:49.323415 4363 tablet_replica.cc:330] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: stopping tablet replica
I20260808 15:55:49.323555 3926 tablet_service.cc:1514] Processing DeleteTablet for tablet d6aadb7742ba4cb8835b9d64d76cf3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:49 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33220
I20260808 15:55:49.323637 4363 raft_consensus.cc:2229] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:49.323856 4363 raft_consensus.cc:2258] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:49.323829 4204 tablet_service.cc:1514] Processing DeleteTablet for tablet d6aadb7742ba4cb8835b9d64d76cf3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:55:49 UTC) from {username='oryx', principal='oryx/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37186
I20260808 15:55:49.324103 4362 tablet_replica.cc:330] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: stopping tablet replica
I20260808 15:55:49.324133 4361 tablet_replica.cc:330] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: stopping tablet replica
I20260808 15:55:49.324208 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 3860
I20260808 15:55:49.324282 4362 raft_consensus.cc:2229] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:55:49.324311 4361 raft_consensus.cc:2229] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:55:49.324512 4361 raft_consensus.cc:2258] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:49.324620 4362 raft_consensus.cc:2258] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:55:49.325013 4363 ts_tablet_manager.cc:1905] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:49.326316 4361 ts_tablet_manager.cc:1905] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:49.330864 4362 ts_tablet_manager.cc:1905] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:55:49.332691 4361 ts_tablet_manager.cc:1918] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:49.332816 4361 log.cc:1196] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/d6aadb7742ba4cb8835b9d64d76cf3be
I20260808 15:55:49.333257 4361 ts_tablet_manager.cc:1939] T d6aadb7742ba4cb8835b9d64d76cf3be P 9a56e55d31744ee7955c1c1b124a8a35: Deleting consensus metadata
I20260808 15:55:49.334441 3774 catalog_manager.cc:4841] TS 9a56e55d31744ee7955c1c1b124a8a35 (127.27.254.1:46733): tablet d6aadb7742ba4cb8835b9d64d76cf3be (table test-table [id=96b1a21c95fb4305a946c1c8b1a70d75]) successfully deleted
I20260808 15:55:49.341065 4363 ts_tablet_manager.cc:1918] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:49.341163 4362 ts_tablet_manager.cc:1918] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:55:49.341289 4362 log.cc:1196] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/d6aadb7742ba4cb8835b9d64d76cf3be
I20260808 15:55:49.341279 4363 log.cc:1196] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/d6aadb7742ba4cb8835b9d64d76cf3be
I20260808 15:55:49.341842 4363 ts_tablet_manager.cc:1939] T d6aadb7742ba4cb8835b9d64d76cf3be P 671e9de0011c464e8b40942eec847f5a: Deleting consensus metadata
I20260808 15:55:49.342178 4362 ts_tablet_manager.cc:1939] T d6aadb7742ba4cb8835b9d64d76cf3be P 19f5277bd5934d64852f6b3e4d6ed19e: Deleting consensus metadata
I20260808 15:55:49.343037 3772 catalog_manager.cc:4841] TS 671e9de0011c464e8b40942eec847f5a (127.27.254.2:35329): tablet d6aadb7742ba4cb8835b9d64d76cf3be (table test-table [id=96b1a21c95fb4305a946c1c8b1a70d75]) successfully deleted
I20260808 15:55:49.343384 3772 catalog_manager.cc:4841] TS 19f5277bd5934d64852f6b3e4d6ed19e (127.27.254.3:40621): tablet d6aadb7742ba4cb8835b9d64d76cf3be (table test-table [id=96b1a21c95fb4305a946c1c8b1a70d75]) successfully deleted
Aug 08 15:55:49 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.1@KRBTEST.COM
I20260808 15:55:49.363744 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:49.329863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.27.254.1:46733)
0808 15:55:49.330198 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:49.330205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:49.344317 (+ 14112us) server_negotiation.cc:407] Connection header received
0808 15:55:49.344635 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:49.344644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:49.344756 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:49.344842 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:49.346631 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.347722 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.349534 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.350210 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.350299 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:49.355313 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:49.355327 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:49.355345 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:49.355386 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:49.360352 (+ 4966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:49.361216 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:49.361226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:49.361229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:49.361369 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:49.361858 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:49.361863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:49.361867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:49.362326 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:49.362612 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:49.363163 (+ 551us) server_negotiation.cc:299] Negotiation successful
0808 15:55:49.363411 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:55:49.365139 3993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:49.462785 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 3860
I20260808 15:55:49.481303 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4002
Aug 08 15:55:49 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.2@KRBTEST.COM
I20260808 15:55:49.498936 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:49.482152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.27.254.2:35329)
0808 15:55:49.482370 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:49.482376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:49.482397 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:55:49.482543 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:49.482549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:49.482640 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:49.482751 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:49.484031 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.485186 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.486596 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.487280 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.487420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:49.491717 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:49.491731 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:49.491750 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:49.491803 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:49.495914 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:49.496677 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:49.496692 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:49.496696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:49.496877 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:49.497288 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:49.497293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:49.497297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:49.497792 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:49.498129 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:49.498310 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 15:55:49.498570 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:55:49.499648 4132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:49.629365 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4002
I20260808 15:55:49.652885 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4141
Aug 08 15:55:49 dist-test-slave-srgv krb5kdc[3743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.254.3@KRBTEST.COM
I20260808 15:55:49.668341 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:49.653702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.27.254.3:40621)
0808 15:55:49.653876 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:49.653882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:49.653928 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:55:49.654150 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:49.654158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:49.654227 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:49.654332 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:49.655570 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.656297 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.657654 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:49.658047 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:49.658193 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:49.662541 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:49.662557 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:49.662573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:49.662612 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:49.665467 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:49.666244 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:49.666253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:49.666258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:49.666416 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:49.666755 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:49.666760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:49.666763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:49.667256 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:49.667555 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:49.667760 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 15:55:49.667982 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:55:49.669042 4271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:49.814136 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4141
I20260808 15:55:49.841758 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 3756
I20260808 15:55:49.842865 3828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:49.967797 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 3756
2026-08-08T15:55:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7045 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:50 dist-test-slave-srgv krb5kdc[4409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:50 dist-test-slave-srgv krb5kdc[4409](info): set up 2 sockets
Aug 08 15:55:50 dist-test-slave-srgv krb5kdc[4409](info): commencing operation
krb5kdc: starting...
W20260808 15:55:51.052412 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:51 dist-test-slave-srgv krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:51.272385 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40885
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:51.594452 4422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:51.594769 4422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:51.594926 4422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:51.602064 4422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:51.602190 4422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:51.602304 4422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:51.602402 4422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:51.602489 4422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:51.612196 4422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:51.613571 4422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:51.614888 4422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:51.626771 4427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:51.626698 4428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:51.626730 4430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:51.627614 4422 server_base.cc:1034] running on GCE node
I20260808 15:55:51.628356 4422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:51.629736 4422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:51.630966 4422 hybrid_clock.cc:648] HybridClock initialized: now 1786204551630913 us; error 40 us; skew 500 ppm
I20260808 15:55:51.631311 4422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T15:55:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1729 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:55:51 dist-test-slave-srgv krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:55:51 dist-test-slave-srgv krb5kdc[4436](info): set up 2 sockets
Aug 08 15:55:51 dist-test-slave-srgv krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260808 15:55:52.808261 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:55:52 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:55:53.027215 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38569
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:53.345691 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:53.345963 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:53.346061 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:53.353018 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:53.353120 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:53.353173 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:53.353233 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:53.353283 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:53.362668 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:53.363884 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:53.365152 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:53.376834 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:53.376775 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:53.376797 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:53.377928 4449 server_base.cc:1034] running on GCE node
I20260808 15:55:53.378690 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:53.380101 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:53.381305 4449 hybrid_clock.cc:648] HybridClock initialized: now 1786204553381269 us; error 31 us; skew 500 ppm
I20260808 15:55:53.381675 4449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:53 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204553, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:53.390764 4449 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:53.393436 4449 webserver.cc:460] Webserver started at http://127.27.254.62:36125/ using document root <none> and password file <none>
I20260808 15:55:53.394716 4449 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:53.394939 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:53.395387 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:53.400296 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "5e3b852ebbb646ae98940471973f7ee7"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "6b1afa81e3741a08dd073192edb19235"
server_key_iv: "05f8e2a54e8301383320ce393429356a"
server_key_version: "encryptionkey@0"
I20260808 15:55:53.401511 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "5e3b852ebbb646ae98940471973f7ee7"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "6b1afa81e3741a08dd073192edb19235"
server_key_iv: "05f8e2a54e8301383320ce393429356a"
server_key_version: "encryptionkey@0"
I20260808 15:55:53.409479 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260808 15:55:53.414605 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:53.416585 4449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260808 15:55:53.416915 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "5e3b852ebbb646ae98940471973f7ee7"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "6b1afa81e3741a08dd073192edb19235"
server_key_iv: "05f8e2a54e8301383320ce393429356a"
server_key_version: "encryptionkey@0"
I20260808 15:55:53.417579 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:53.441195 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:53.445051 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:53.445338 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:53.465988 4449 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:38569
I20260808 15:55:53.465991 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:38569 every 8 connection(s)
I20260808 15:55:53.468232 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:53.470412 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4449
I20260808 15:55:53.470666 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:55:53.471619 28664 external_mini_cluster.cc:1428] Setting key 4130d0abc95e3022f72d1bb8c79bb81f
I20260808 15:55:53.475163 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:55:53.486179 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Bootstrap starting.
Aug 08 15:55:53 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:53.489854 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Neither blocks nor log segments found. Creating new log.
I20260808 15:55:53.490911 4516 log.cc:824] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:53.494213 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: No bootstrap required, opened a new log
I20260808 15:55:53.495929 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:53.473779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.27.254.62:38569)
0808 15:55:53.474669 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:53.474688 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:53.474747 (+ 59us) server_negotiation.cc:407] Connection header received
0808 15:55:53.477637 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:53.477670 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:53.478132 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:53.478552 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:53.480037 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:53.481563 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:53.482881 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:53.483572 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:53.483803 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:53.488152 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:53.488160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:53.488174 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:53.488213 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:53.491406 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:53.492132 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:53.492139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:53.492145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:53.492305 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:53.492766 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:53.492771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:53.492774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:53.493479 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:53.493757 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:53.494157 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 15:55:53.494596 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":355,"threads_started":1}
I20260808 15:55:53.502568 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:53.502874 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:55:53.502942 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3b852ebbb646ae98940471973f7ee7, State: Initialized, Role: FOLLOWER
I20260808 15:55:53.503451 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:53.503654 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:53.503749 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:53.503865 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:55:53.506199 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:53.506633 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e3b852ebbb646ae98940471973f7ee7; no voters:
I20260808 15:55:53.507189 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:55:53.507349 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:55:53.508038 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e3b852ebbb646ae98940471973f7ee7, State: Running, Role: LEADER
I20260808 15:55:53.508565 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:53.509168 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:53.511654 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e3b852ebbb646ae98940471973f7ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } } }
I20260808 15:55:53.512130 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:53.512926 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e3b852ebbb646ae98940471973f7ee7. Latest consensus state: current_term: 1 leader_uuid: "5e3b852ebbb646ae98940471973f7ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } } }
I20260808 15:55:53.513276 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:53.514605 4528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:53.518867 4528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:53.528012 4528 catalog_manager.cc:1347] Generated new cluster ID: 29e151fce5b74e389353c7775d1f1c11
I20260808 15:55:53.528122 4528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:53.544303 4528 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:55:53.545650 4528 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:53.551436 4528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Generated new TSK 0
I20260808 15:55:53.552203 4528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:55:53.605438 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38569
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:53.932960 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:53.933269 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:53.933482 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:53.940618 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:53.940773 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:53.940968 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:53.952154 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:53.953909 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:53.955197 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:53.967942 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:53.968029 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:53.968006 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:53.969247 4542 server_base.cc:1034] running on GCE node
I20260808 15:55:53.969858 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:53.970786 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:53.971979 4542 hybrid_clock.cc:648] HybridClock initialized: now 1786204553971945 us; error 26 us; skew 500 ppm
I20260808 15:55:53.972313 4542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:53 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204553, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:53.975333 4542 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:55:53.976711 4542 webserver.cc:460] Webserver started at http://127.27.254.1:39243/ using document root <none> and password file <none>
I20260808 15:55:53.977399 4542 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:53.977607 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:53.977917 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:53.980731 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "d6cad423ebef44949aa627370dda6381"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "a911a2280512a0a6400aa8bf9e451658"
server_key_iv: "a63b99840af6f6162c290faefd5782bf"
server_key_version: "encryptionkey@0"
I20260808 15:55:53.981665 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "d6cad423ebef44949aa627370dda6381"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "a911a2280512a0a6400aa8bf9e451658"
server_key_iv: "a63b99840af6f6162c290faefd5782bf"
server_key_version: "encryptionkey@0"
I20260808 15:55:53.986651 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:55:53.990034 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:53.991330 4542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:55:53.991508 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "d6cad423ebef44949aa627370dda6381"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "a911a2280512a0a6400aa8bf9e451658"
server_key_iv: "a63b99840af6f6162c290faefd5782bf"
server_key_version: "encryptionkey@0"
I20260808 15:55:53.991709 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:54.011709 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:54.015677 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:54.015957 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:54.016906 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:54.018342 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:54.018445 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:54.018591 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:54.018663 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:54.051383 4542 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:39165
I20260808 15:55:54.051435 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:39165 every 8 connection(s)
I20260808 15:55:54.052639 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:54.058727 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4542
I20260808 15:55:54.058938 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:55:54.059561 28664 external_mini_cluster.cc:1428] Setting key 833b88022f388a8c6a208295b46f3c72
Aug 08 15:55:54 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204553, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:54.072397 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:54.055234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:45971 (local address 127.27.254.62:38569)
0808 15:55:54.055662 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:54.055667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:54.056531 (+ 864us) server_negotiation.cc:407] Connection header received
0808 15:55:54.058421 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:54.058427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:54.058512 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:54.058635 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:54.060201 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.061144 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.062723 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.063109 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.063242 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:54.066680 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:54.066685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:54.066689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:54.066719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:54.069219 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:54.070146 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:54.070150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:54.070152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:54.070274 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:54.070763 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:54.070766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:54.070768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:54.071002 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:54.071109 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:54.071948 (+ 839us) server_negotiation.cc:299] Negotiation successful
0808 15:55:54.072101 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":237,"threads_started":1}
I20260808 15:55:54.074158 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:54.055665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:45971)
0808 15:55:54.056364 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:54.056406 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:54.058140 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:54.058902 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:54.058911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:54.059455 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:54.060033 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:54.060051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.061321 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.061325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:54.062578 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:54.062591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.063340 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.063349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:54.063582 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:54.064249 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:54.064275 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:54.066455 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:54.069357 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:54.069368 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:54.069385 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:54.070027 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:54.070386 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:54.070393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:54.070399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:54.070650 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:54.071133 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:54.071147 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:54.071570 (+ 423us) client_negotiation.cc:769] Sending connection context
0808 15:55:54.071869 (+ 299us) client_negotiation.cc:240] Negotiation successful
0808 15:55:54.072716 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":266,"threads_started":1}
I20260808 15:55:54.075618 4670 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38569
I20260808 15:55:54.075932 4670 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:54.076809 4670 heartbeater.cc:511] Master 127.27.254.62:38569 requested a full tablet report, sending...
I20260808 15:55:54.079005 4481 ts_manager.cc:194] Registered new tserver with Master: d6cad423ebef44949aa627370dda6381 (127.27.254.1:39165)
I20260808 15:55:54.080453 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:45971
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:55:54.109053 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38569
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:54.428774 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:54.429044 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:54.429196 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:54.436506 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:54.436614 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:54.436774 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:54.446611 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:54.447927 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:54.449241 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:54.462082 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:54.462050 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:54.462113 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:54.463341 4676 server_base.cc:1034] running on GCE node
I20260808 15:55:54.463949 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:54.464797 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:54.466048 4676 hybrid_clock.cc:648] HybridClock initialized: now 1786204554465979 us; error 49 us; skew 500 ppm
I20260808 15:55:54.466435 4676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:54 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204554, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:54.469812 4676 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:55:54.478523 4676 webserver.cc:460] Webserver started at http://127.27.254.2:40159/ using document root <none> and password file <none>
I20260808 15:55:54.479206 4676 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:54.479378 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:54.479688 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:54.482595 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "a9874510415046519f14710132c8a582"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "20deb9a1c71cd316723570cec175ebc1"
server_key_iv: "b74da78105ff1808f1f5d25a386b6ac2"
server_key_version: "encryptionkey@0"
I20260808 15:55:54.483381 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "a9874510415046519f14710132c8a582"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "20deb9a1c71cd316723570cec175ebc1"
server_key_iv: "b74da78105ff1808f1f5d25a386b6ac2"
server_key_version: "encryptionkey@0"
I20260808 15:55:54.488430 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 15:55:54.491909 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:54.493135 4676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:55:54.493309 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "a9874510415046519f14710132c8a582"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "20deb9a1c71cd316723570cec175ebc1"
server_key_iv: "b74da78105ff1808f1f5d25a386b6ac2"
server_key_version: "encryptionkey@0"
I20260808 15:55:54.493561 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:54.516248 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:54.520804 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:54.521106 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:54.522154 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:54.523528 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:54.523628 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:54.523775 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:54.523846 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:54.556807 4676 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:34471
I20260808 15:55:54.556897 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:34471 every 8 connection(s)
I20260808 15:55:54.558106 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:54.559388 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4676
I20260808 15:55:54.559567 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:55:54.560009 28664 external_mini_cluster.cc:1428] Setting key 0af4938bed36f93c581f5ae4eb5fc1eb
Aug 08 15:55:54 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204554, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:54.579224 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:54.560850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:34149 (local address 127.27.254.62:38569)
0808 15:55:54.561059 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:54.561064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:54.562401 (+ 1337us) server_negotiation.cc:407] Connection header received
0808 15:55:54.564216 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:54.564225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:54.564317 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:54.564431 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:54.565914 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.567058 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.568534 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.569158 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.569266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:54.573344 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:54.573349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:54.573352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:54.573382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:54.576041 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:54.576778 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:54.576782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:54.576784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:54.576904 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:54.577348 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:54.577352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:54.577355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:54.577666 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:54.577827 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:54.578608 (+ 781us) server_negotiation.cc:299] Negotiation successful
0808 15:55:54.578858 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:55:54.580160 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:54.561350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:34149)
0808 15:55:54.562228 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:54.562285 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:54.563947 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:54.564624 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:54.564632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:54.565113 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:54.565756 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:54.565774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.567187 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.567191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:54.568391 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:54.568402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:54.569401 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:54.569437 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:54.569751 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:54.570493 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:54.570526 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:54.573105 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:54.576188 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:54.576196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:54.576222 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:54.576633 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:54.577018 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:54.577023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:54.577027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:54.577219 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:54.577816 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:54.577824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:54.578245 (+ 421us) client_negotiation.cc:769] Sending connection context
0808 15:55:54.578504 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 15:55:54.579020 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":355,"threads_started":1}
I20260808 15:55:54.581784 4804 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38569
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
I20260808 15:55:54.582108 4804 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:54.582820 4804 heartbeater.cc:511] Master 127.27.254.62:38569 requested a full tablet report, sending...
I20260808 15:55:54.584379 4481 ts_manager.cc:194] Registered new tserver with Master: a9874510415046519f14710132c8a582 (127.27.254.2:34471)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
I20260808 15:55:54.585475 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:34149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:55:54.612262 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38569
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:54.935431 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:54.935689 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:54.935808 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:54.942945 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:54.943073 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:54.943251 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:54.953751 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:54.955085 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:54.956337 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:54.969775 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:54.969815 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:54.969947 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:54.970804 4809 server_base.cc:1034] running on GCE node
I20260808 15:55:54.971413 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:54.972462 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:54.973722 4809 hybrid_clock.cc:648] HybridClock initialized: now 1786204554973661 us; error 44 us; skew 500 ppm
I20260808 15:55:54.973996 4809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:54 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204554, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:54.976863 4809 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:55:54.978327 4809 webserver.cc:460] Webserver started at http://127.27.254.3:33893/ using document root <none> and password file <none>
I20260808 15:55:54.978996 4809 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:54.979138 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:54.979434 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:55:54.982303 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "7135e392e8834e669be4c9f2d947a254"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "ac5e7f283d19916a2918aab3c47fef39"
server_key_iv: "bb7ee3b04d09b157ee4a1c2f75d23c82"
server_key_version: "encryptionkey@0"
I20260808 15:55:54.983064 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "7135e392e8834e669be4c9f2d947a254"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "ac5e7f283d19916a2918aab3c47fef39"
server_key_iv: "bb7ee3b04d09b157ee4a1c2f75d23c82"
server_key_version: "encryptionkey@0"
I20260808 15:55:54.988042 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:55:54.991439 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:54.992724 4809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:54.992905 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "7135e392e8834e669be4c9f2d947a254"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "ac5e7f283d19916a2918aab3c47fef39"
server_key_iv: "bb7ee3b04d09b157ee4a1c2f75d23c82"
server_key_version: "encryptionkey@0"
I20260808 15:55:54.993093 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:55.007797 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:55.012073 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:55.012377 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:55.013274 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:55.014546 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:55.014619 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:55.014791 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:55.014845 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:55.045553 4809 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:43565
I20260808 15:55:55.045574 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:43565 every 8 connection(s)
I20260808 15:55:55.046814 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:55.053320 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4809
I20260808 15:55:55.053516 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:55:55.054088 28664 external_mini_cluster.cc:1428] Setting key 867455021733bb4003328099ee55c513
Aug 08 15:55:55 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204554, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:55:55.066452 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:55.049281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:51941 (local address 127.27.254.62:38569)
0808 15:55:55.049514 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:55.049519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:55.050640 (+ 1121us) server_negotiation.cc:407] Connection header received
0808 15:55:55.052664 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:55.052670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:55.052791 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:55.052907 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:55.054640 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.055742 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.057659 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.058043 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.058153 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:55.061327 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:55.061332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:55.061335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:55.061368 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:55.063801 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.064471 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.064475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.064477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.064599 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.064948 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.064950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.064953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.065229 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:55.065347 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:55.066016 (+ 669us) server_negotiation.cc:299] Negotiation successful
0808 15:55:55.066173 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:55:55.067368 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:55.049796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:51941)
0808 15:55:55.050484 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:55.050520 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:55.052393 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:55.053151 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:55.053159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:55.053834 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:55.054449 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:55.054468 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.055909 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.055915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:55.057521 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:55.057533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.058236 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.058240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:55.058420 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:55.059021 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:55.059044 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:55.061138 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:55:55.063921 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:55.063927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:55.063940 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:55.064352 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:55.064686 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:55:55.064689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:55:55.064691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:55:55.064855 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:55:55.065341 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:55:55.065347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:55:55.065703 (+ 356us) client_negotiation.cc:769] Sending connection context
0808 15:55:55.065917 (+ 214us) client_negotiation.cc:240] Negotiation successful
0808 15:55:55.066271 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":276,"threads_started":1}
I20260808 15:55:55.068771 4937 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38569
I20260808 15:55:55.069072 4937 heartbeater.cc:464] Registering TS with master...
I20260808 15:55:55.069877 4937 heartbeater.cc:511] Master 127.27.254.62:38569 requested a full tablet report, sending...
I20260808 15:55:55.071204 4480 ts_manager.cc:194] Registered new tserver with Master: 7135e392e8834e669be4c9f2d947a254 (127.27.254.3:43565)
I20260808 15:55:55.072026 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:51941
I20260808 15:55:55.081526 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:55:55.083412 4670 heartbeater.cc:503] Master 127.27.254.62:38569 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab.
Entry for principal oryx/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab.
WARNING: no policy specified for oryx/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab.
Entry for principal oryx/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab.
WARNING: no policy specified for oryx/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab.
Entry for principal oryx/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab.
I20260808 15:55:55.213184 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4542
Aug 08 15:55:55 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:55:55.230152 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:55.214443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.27.254.1:39165)
0808 15:55:55.215151 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:55.215159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:55.215186 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:55:55.215251 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:55.215258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:55.215471 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:55.215611 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:55.217001 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.218143 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.219545 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.220148 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.220260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:55.224446 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:55.224459 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:55.224473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:55.224509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:55.227250 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.227972 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.227979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.227982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.228104 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.228491 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.228497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.228500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.228836 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:55.229165 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:55.229492 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 15:55:55.229773 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":377,"threads_started":1}
I20260808 15:55:55.231477 4665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:55.315022 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4542
I20260808 15:55:55.330978 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4676
Aug 08 15:55:55 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:55:55.347051 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:55.331886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.27.254.2:34471)
0808 15:55:55.332388 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:55.332395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:55.332419 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:55:55.332482 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:55.332488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:55.332690 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:55.332864 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:55.334141 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.335080 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.336369 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.336924 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.337065 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:55.341343 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:55.341352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:55.341367 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:55.341450 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:55.344128 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.344808 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.344815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.344817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.344967 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.345332 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.345339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.345342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.345830 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:55.346170 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:55.346456 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 15:55:55.346719 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260808 15:55:55.348425 4799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:55.429450 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4676
I20260808 15:55:55.446148 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4809
Aug 08 15:55:55 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:55:55.462695 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:55.447121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.27.254.3:43565)
0808 15:55:55.447746 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:55.447752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:55.447781 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:55:55.447842 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:55.447848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:55.448088 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:55.448283 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:55.449708 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.450745 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.452044 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:55.452613 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:55.452753 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:55.457008 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:55.457018 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:55.457033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:55.457064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:55.459704 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.460371 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.460378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.460381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.460502 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:55.460885 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:55.460890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:55.460893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:55.461249 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:55.461661 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:55.462096 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 15:55:55.462374 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":317,"threads_started":1}
I20260808 15:55:55.463951 4932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:55.549791 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4809
I20260808 15:55:55.566402 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4449
I20260808 15:55:55.567700 4511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:55:55.671487 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4449
I20260808 15:55:55.691762 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38569
--webserver_interface=127.27.254.62
--webserver_port=36125
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:55:56.018589 4954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:56.018859 4954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:56.018975 4954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:56.026132 4954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:55:56.026240 4954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:56.026294 4954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:56.026347 4954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:55:56.026391 4954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:55:56.036091 4954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:56.037375 4954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:56.038669 4954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:56.052145 4959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:56.052172 4962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:56.052213 4960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:56.052891 4954 server_base.cc:1034] running on GCE node
I20260808 15:55:56.053808 4954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:56.055192 4954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:56.056406 4954 hybrid_clock.cc:648] HybridClock initialized: now 1786204556056358 us; error 29 us; skew 500 ppm
I20260808 15:55:56.056769 4954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:56 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204556, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:56.059894 4954 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:55:56.061491 4954 webserver.cc:460] Webserver started at http://127.27.254.62:36125/ using document root <none> and password file <none>
I20260808 15:55:56.062157 4954 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:56.062268 4954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:56.067572 4954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 15:55:56.070230 4969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:56.071527 4954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:55:56.071694 4954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "5e3b852ebbb646ae98940471973f7ee7"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "6b1afa81e3741a08dd073192edb19235"
server_key_iv: "05f8e2a54e8301383320ce393429356a"
server_key_version: "encryptionkey@0"
I20260808 15:55:56.072659 4954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:56.091815 4954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:56.095650 4954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:56.095912 4954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:56.113924 4954 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:38569
I20260808 15:55:56.113945 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:38569 every 8 connection(s)
I20260808 15:55:56.115419 4954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:55:56.122159 5021 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:55:56.123061 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 4954
I20260808 15:55:56.123843 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:39165
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:38569
--webserver_port=39243
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:55:56.125556 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Bootstrap starting.
I20260808 15:55:56.144418 5021 log.cc:824] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:55:56.153330 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:55:56.153960 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Bootstrap complete.
I20260808 15:55:56.163053 5021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:56.163898 5021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3b852ebbb646ae98940471973f7ee7, State: Initialized, Role: FOLLOWER
I20260808 15:55:56.164548 5021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:56.164801 5021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:55:56.164921 5021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:55:56.165099 5021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:55:56.168871 5021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:56.169358 5021 leader_election.cc:302] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e3b852ebbb646ae98940471973f7ee7; no voters:
I20260808 15:55:56.170096 5021 leader_election.cc:288] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:55:56.170243 5025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:55:56.171631 5025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [term 2 LEADER]: Becoming Leader. State: Replica: 5e3b852ebbb646ae98940471973f7ee7, State: Running, Role: LEADER
I20260808 15:55:56.172242 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } }
I20260808 15:55:56.172422 5021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:55:56.174966 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e3b852ebbb646ae98940471973f7ee7. Latest consensus state: current_term: 2 leader_uuid: "5e3b852ebbb646ae98940471973f7ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } } }
I20260808 15:55:56.175328 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:56.175844 5026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e3b852ebbb646ae98940471973f7ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3b852ebbb646ae98940471973f7ee7" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38569 } } }
I20260808 15:55:56.176156 5026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:55:56.178246 5033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:55:56.185359 5033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:55:56.187247 5033 catalog_manager.cc:1259] Loaded cluster ID: 29e151fce5b74e389353c7775d1f1c11
I20260808 15:55:56.187443 5033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:55:56.192729 5033 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:55:56.195055 5033 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e3b852ebbb646ae98940471973f7ee7: Loaded TSK: 0
I20260808 15:55:56.197067 5033 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:55:56.473646 5023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:56.474013 5023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:56.474206 5023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:56.481976 5023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:56.482138 5023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:56.482308 5023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:55:56.492125 5023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.1.keytab
--principal=oryx/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:39165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=39243
--tserver_master_addrs=127.27.254.62:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:56.493577 5023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:56.494927 5023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:56.509689 5047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:56.509877 5050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:56.510459 5048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:56.510670 5023 server_base.cc:1034] running on GCE node
I20260808 15:55:56.511407 5023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:56.512511 5023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:56.513870 5023 hybrid_clock.cc:648] HybridClock initialized: now 1786204556513790 us; error 50 us; skew 500 ppm
I20260808 15:55:56.514272 5023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:56 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204556, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:56.518128 5023 init.cc:377] Logged in from keytab as oryx/127.27.254.1@KRBTEST.COM (short username oryx)
I20260808 15:55:56.519954 5023 webserver.cc:460] Webserver started at http://127.27.254.1:39243/ using document root <none> and password file <none>
I20260808 15:55:56.520807 5023 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:56.520955 5023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:56.527918 5023 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 15:55:56.531298 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:56.532984 5023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:55:56.533174 5023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "d6cad423ebef44949aa627370dda6381"
format_stamp: "Formatted at 2026-08-08 15:55:53 on dist-test-slave-srgv"
server_key: "a911a2280512a0a6400aa8bf9e451658"
server_key_iv: "a63b99840af6f6162c290faefd5782bf"
server_key_version: "encryptionkey@0"
I20260808 15:55:56.534616 5023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:56.559346 5023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:56.564432 5023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:56.564728 5023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:56.565937 5023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:56.567561 5023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:56.567667 5023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:56.567770 5023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:56.567838 5023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:56.602012 5023 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:39165
I20260808 15:55:56.602067 5170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:39165 every 8 connection(s)
I20260808 15:55:56.603525 5023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:55:56.607529 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5023
I20260808 15:55:56.608167 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:34471
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:38569
--webserver_port=40159
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:55:56 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:56 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:55:56.942762 5175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:56.943030 5175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:56.943185 5175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:56.950630 5175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:56.950809 5175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:56.951001 5175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:55:56.960882 5175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.2.keytab
--principal=oryx/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:34471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=40159
--tserver_master_addrs=127.27.254.62:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:56.962749 5175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:56.964111 5175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:56.986147 5181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:56.986786 5182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:56.987831 5175 server_base.cc:1034] running on GCE node
W20260808 15:55:56.988142 5184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:56.989459 5175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:56.991230 5175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:56.992537 5175 hybrid_clock.cc:648] HybridClock initialized: now 1786204556992451 us; error 69 us; skew 500 ppm
I20260808 15:55:56.993120 5175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:56 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204556, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:56.998380 5175 init.cc:377] Logged in from keytab as oryx/127.27.254.2@KRBTEST.COM (short username oryx)
I20260808 15:55:57.000277 5175 webserver.cc:460] Webserver started at http://127.27.254.2:40159/ using document root <none> and password file <none>
I20260808 15:55:57.001014 5175 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:57.001165 5175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:57.007776 5175 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260808 15:55:57.010803 5191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:57.012333 5175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:55:57.012503 5175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "a9874510415046519f14710132c8a582"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "20deb9a1c71cd316723570cec175ebc1"
server_key_iv: "b74da78105ff1808f1f5d25a386b6ac2"
server_key_version: "encryptionkey@0"
I20260808 15:55:57.013496 5175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:57.033116 5175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:57.037622 5175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:57.037902 5175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:57.038991 5175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:57.040593 5175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:57.040685 5175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:57.040832 5175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:57.040911 5175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:57.088037 5175 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:34471
I20260808 15:55:57.088094 5303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:34471 every 8 connection(s)
I20260808 15:55:57.089636 5175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:55:57.093626 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5175
I20260808 15:55:57.094246 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:43565
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:38569
--webserver_port=33893
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:55:57.701463 5308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:55:57.701799 5308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:55:57.701961 5308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:55:57.709467 5308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:55:57.709621 5308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:55:57.709810 5308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:55:57.719646 5308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/krb5kdc/oryx_127.27.254.3.keytab
--principal=oryx/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:43565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=33893
--tserver_master_addrs=127.27.254.62:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:55:57.720971 5308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:55:57.722343 5308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:55:57.736703 5315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:57.736796 5314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:55:57.737655 5317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:55:57.739019 5308 server_base.cc:1034] running on GCE node
I20260808 15:55:57.739790 5308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:55:57.740962 5308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:55:57.742509 5308 hybrid_clock.cc:648] HybridClock initialized: now 1786204557742382 us; error 123 us; skew 500 ppm
I20260808 15:55:57.742939 5308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204557, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:55:57.746541 5308 init.cc:377] Logged in from keytab as oryx/127.27.254.3@KRBTEST.COM (short username oryx)
I20260808 15:55:57.748374 5308 webserver.cc:460] Webserver started at http://127.27.254.3:33893/ using document root <none> and password file <none>
I20260808 15:55:57.749102 5308 fs_manager.cc:359] Metadata directory not provided
I20260808 15:55:57.749248 5308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:55:57.755635 5308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260808 15:55:57.758316 5324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:57.760840 5308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260808 15:55:57.761009 5308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "7135e392e8834e669be4c9f2d947a254"
format_stamp: "Formatted at 2026-08-08 15:55:54 on dist-test-slave-srgv"
server_key: "ac5e7f283d19916a2918aab3c47fef39"
server_key_iv: "bb7ee3b04d09b157ee4a1c2f75d23c82"
server_key_version: "encryptionkey@0"
I20260808 15:55:57.762053 5308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:55:57.776794 5308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:55:57.781266 5308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:55:57.781572 5308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:55:57.782639 5308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:55:57.784183 5308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:55:57.784304 5308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:57.784435 5308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:55:57.784535 5308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:55:57.836128 5308 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:43565
I20260808 15:55:57.836282 5436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:43565 every 8 connection(s)
I20260808 15:55:57.837971 5308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:55:57.845539 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5308
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:57.890419 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:57.846734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.27.254.62:38569)
0808 15:55:57.866288 (+ 19554us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:57.866293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:57.866309 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:55:57.866353 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:57.866358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:57.866428 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:57.866527 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:57.869392 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.870723 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.872945 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.873740 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.873871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.877847 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:55:57.877864 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:55:57.877899 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:55:57.877978 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:55:57.883778 (+ 5800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:57.884773 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:57.884788 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:57.884803 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:57.885139 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:55:57.885696 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:55:57.885704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:55:57.885709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:55:57.886700 (+ 991us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:55:57.887376 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0808 15:55:57.888069 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 15:55:57.888797 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19408,"thread_start_us":19272,"threads_started":1}
W20260808 15:55:57.922139 5174 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.301s user 0.008s sys 0.006s
I20260808 15:55:57.925127 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:56.606631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:37433)
0808 15:55:56.610035 (+ 3404us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:56.610082 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:56.611829 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:56.613539 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:56.613546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:56.614065 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:56.614612 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:56.614629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:56.617118 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:56.617122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:56.618401 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:56.618413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:56.620028 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:56.620033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:56.620234 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:56.620928 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:56.620951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:57.923412 (+1302461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3099,"thread_start_us":2895,"threads_started":1}
W20260808 15:55:57.927271 5173 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:56.606609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:37433 (local address 127.27.254.62:38569)
0808 15:55:56.607436 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:56.607452 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:56.610244 (+ 2792us) server_negotiation.cc:407] Connection header received
0808 15:55:56.612467 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:56.612488 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:56.612955 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:56.613384 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:56.614843 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:56.616949 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:56.618559 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:56.619734 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:56.619925 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.926798 (+1306873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:37433: BlockingRecv error: recv got EOF from 127.27.254.1:37433 (error 108)
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":304,"threads_started":1}
W20260808 15:55:57.927649 5171 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:38569 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.254.62:38569: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:57.945139 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:57.928578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:43931)
0808 15:55:57.928946 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:57.928974 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:57.929092 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:57.930031 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:57.930039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:57.930744 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:57.931070 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.931086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.933200 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.933209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.935528 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.935546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.936961 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.936970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.937257 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.938268 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:57.938304 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:57.944665 (+ 6361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260808 15:55:57.958707 5173 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:57.928780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:43931 (local address 127.27.254.62:38569)
0808 15:55:57.929335 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:57.929345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:57.929371 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:55:57.929470 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:57.929479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:57.929585 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:57.929727 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:57.931307 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.932986 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.935726 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.936708 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.936890 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.958299 (+ 21409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:43931: BlockingRecv error: recv got EOF from 127.27.254.1:43931 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":391}
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:57 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:57.971163 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:57.946319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:55743)
0808 15:55:57.952186 (+ 5867us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:57.952214 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:57.952367 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:57.954877 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:57.954884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:57.955506 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:57.955819 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.955835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.958056 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.958065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.960810 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.960836 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.962644 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.962653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.962939 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.963886 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:57.963921 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:57.970659 (+ 6738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5692,"thread_start_us":11204,"threads_started":1}
W20260808 15:55:57.971459 5442 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:57.954322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:55743 (local address 127.27.254.62:38569)
0808 15:55:57.954526 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:57.954531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:57.954547 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:55:57.954617 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:57.954622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:57.954723 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:57.954823 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:57.956086 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.957825 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.961074 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.962148 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.962290 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.971027 (+ 8737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:55743: BlockingRecv error: recv got EOF from 127.27.254.1:55743 (error 108)
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":54}
W20260808 15:55:57.971959 5171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:55:58.378511 5307 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.263s user 0.003s sys 0.023s
W20260808 15:55:58.379841 5306 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:57.092810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:45037 (local address 127.27.254.62:38569)
0808 15:55:57.093290 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:57.093297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:57.101661 (+ 8364us) server_negotiation.cc:407] Connection header received
0808 15:55:57.104800 (+ 3139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:57.104809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:57.104905 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:57.104989 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:57.107575 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.109282 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.112074 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.113083 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.113264 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.379517 (+1266253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:45037: BlockingRecv error: recv got EOF from 127.27.254.2:45037 (error 108)
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260808 15:55:58.380393 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:57.093976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:45037)
0808 15:55:57.101463 (+ 7487us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:57.101527 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:57.104455 (+ 2928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:57.105652 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:57.105667 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:57.106474 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:57.107348 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.107375 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.109519 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.109528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.111863 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.111883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.113797 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.113808 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.114278 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.115429 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:57.115476 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:58.379290 (+1263814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7022,"thread_start_us":6678,"threads_started":1}
W20260808 15:55:58.381336 5304 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:38569 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.254.62:38569: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:58.400414 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:58.382247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:35219)
0808 15:55:58.382610 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:58.382637 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:58.382816 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:58.383188 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:58.383196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:58.383872 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:58.384221 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.384239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.385391 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.385399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.387737 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.387760 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.389202 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.389217 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.389533 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.395003 (+ 5470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:58.395054 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:58.400087 (+ 5033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193}
W20260808 15:55:58.400629 5306 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:58.382240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:35219 (local address 127.27.254.62:38569)
0808 15:55:58.382397 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:58.382401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:58.382724 (+ 323us) server_negotiation.cc:407] Connection header received
0808 15:55:58.382954 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:58.382957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:58.383017 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:58.383100 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:58.384430 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.385222 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.387955 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.388498 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.388650 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.400352 (+ 11702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:35219: BlockingRecv error: recv got EOF from 127.27.254.2:35219 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:55:58.412885 5306 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:58.401877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:41291 (local address 127.27.254.62:38569)
0808 15:55:58.402052 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:58.402057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:58.402075 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:55:58.402316 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:58.402325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:58.402481 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:58.402743 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:58.403467 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.405212 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.406601 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.407585 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.407778 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.412600 (+ 4822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:41291: BlockingRecv error: recv got EOF from 127.27.254.2:41291 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:55:58.412881 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:58.401755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:41291)
0808 15:55:58.401966 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:58.401985 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:58.402125 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:58.402801 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:58.402806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:58.403139 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:58.403297 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.403305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.405377 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.405380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.406394 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.406404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.407763 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.407767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.407889 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.408477 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:58.408501 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:58.412450 (+ 3949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260808 15:55:58.413247 5304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:55:58.981802 5440 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.115s user 0.004s sys 0.014s
I20260808 15:55:58.983088 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:58.972551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:39991)
0808 15:55:58.973155 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:58.973180 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:58.973317 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:58.973769 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:58.973774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:58.974233 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:58.974379 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.974388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.975393 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.975398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.976802 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.976818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.978163 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.978169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.978351 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.978960 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:58.978984 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:58.982758 (+ 3774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":340,"threads_started":1}
W20260808 15:55:58.983104 5439 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:57.841734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:46305 (local address 127.27.254.62:38569)
0808 15:55:57.855147 (+ 13413us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:57.855157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:57.855191 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:55:57.857136 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:57.857142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:57.857222 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:57.857318 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:57.860247 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.861592 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.863996 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.864691 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.864783 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.982823 (+1118040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:46305: BlockingRecv error: recv got EOF from 127.27.254.3:46305 (error 108)
Metrics: {"server-negotiator.queue_time_us":13227,"thread_start_us":13044,"threads_started":1}
W20260808 15:55:58.983624 5446 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:58.972635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:39991 (local address 127.27.254.62:38569)
0808 15:55:58.973457 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:58.973462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:58.973477 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:55:58.973530 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:58.973533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:58.973598 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:58.973727 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:58.974524 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.975246 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.976987 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.977770 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.977894 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.982994 (+ 5100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:39991: BlockingRecv error: recv got EOF from 127.27.254.1:39991 (error 108)
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":555,"threads_started":1}
I20260808 15:55:58.983790 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:57.842803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:46305)
0808 15:55:57.851510 (+ 8707us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:57.851582 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:57.856668 (+ 5086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:57.858472 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:57.858484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:57.859267 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:57.860019 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.860043 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.861776 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.861782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.863832 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:57.863853 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:57.865137 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:57.865144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:57.865466 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:57.866871 (+ 1405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:57.866899 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:58.982657 (+1115758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8196,"thread_start_us":10463,"threads_started":1}
W20260808 15:55:58.984609 5437 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:38569 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.254.62:38569: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:58.992928 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:58.985099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:47351)
0808 15:55:58.985267 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:58.985287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:58.985395 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:58.985717 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:58.985720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:58.986017 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:58.986185 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.986193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.987053 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.987057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.987972 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.987983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.988680 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.988686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.988847 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.989480 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:58.989505 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:58.992736 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 15:55:58.993096 5446 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:58.985209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:47351 (local address 127.27.254.62:38569)
0808 15:55:58.985460 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:58.985466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:58.985485 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:55:58.985533 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:58.985537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:58.985602 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:58.985678 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:58.986299 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.986942 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.988119 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.988520 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.988604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.992897 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:47351: BlockingRecv error: recv got EOF from 127.27.254.3:47351 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:58 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:59.000735 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:58.993558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:42657)
0808 15:55:58.993694 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:58.993710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:58.993795 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:58.994012 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:58.994014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:58.994268 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:58.994397 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.994404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.995277 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.995281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.996181 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:58.996191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.996791 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.996795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:58.996909 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:58.997352 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:58.997367 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:59.000460 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 15:55:59.000947 5446 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:58.993659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:42657 (local address 127.27.254.62:38569)
0808 15:55:58.993812 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:58.993816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:58.993828 (+ 12us) server_negotiation.cc:407] Connection header received
0808 15:55:58.993863 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:58.993866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:58.993910 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:58.993982 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:58.994507 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.995168 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.996291 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:58.996670 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:58.996761 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:59.000678 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:42657: BlockingRecv error: recv got EOF from 127.27.254.3:42657 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 15:55:59.000981 5437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 15:55:59 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:59 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:59.422863 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:59.413939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:49191)
0808 15:55:59.414531 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:59.414547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:59.414660 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:59.414961 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:59.414964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:59.415417 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:59.415556 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:59.415563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.416727 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.416734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:59.417849 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:59.417860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.418605 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.418612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:59.418773 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:59.419321 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:59.419337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:59.422525 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":368,"threads_started":1}
W20260808 15:55:59.422969 5446 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:59.414151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:49191 (local address 127.27.254.62:38569)
0808 15:55:59.414333 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:59.414337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:59.414620 (+ 283us) server_negotiation.cc:407] Connection header received
0808 15:55:59.414759 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:59.414762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:59.414832 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:59.414915 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:59.415732 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.416580 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.417990 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.418415 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.418546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:59.422742 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:49191: BlockingRecv error: recv got EOF from 127.27.254.2:49191 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 15:55:59 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:55:59 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:55:59.993036 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:55:59.984089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:39181)
0808 15:55:59.984613 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 15:55:59.984629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:55:59.984753 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:55:59.985056 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:55:59.985059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:55:59.985474 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:55:59.985657 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:59.985666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.986952 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.986958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:59.988007 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:55:59.988021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.988702 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.988707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:55:59.988826 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:59.989371 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:55:59.989389 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:55:59.992790 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
W20260808 15:55:59.993257 5449 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:55:59.984181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:39181 (local address 127.27.254.62:38569)
0808 15:55:59.984655 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 15:55:59.984659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:55:59.984679 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:55:59.984861 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:55:59.984865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:55:59.984923 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:55:59.985013 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:55:59.985829 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.986802 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.988150 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:55:59.988564 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:55:59.988666 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:55:59.992995 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:39181: BlockingRecv error: recv got EOF from 127.27.254.1:39181 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
Aug 08 15:56:00 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:00 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:00.010434 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:00.001580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:42227)
0808 15:56:00.002152 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:00.002168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:00.002273 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:00.002531 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:00.002534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:00.002927 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:00.003066 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:00.003074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.004008 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.004012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:00.004951 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:00.004962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.005705 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.005709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:00.005883 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:00.006501 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:00.006524 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:00.010119 (+ 3595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":329,"threads_started":1}
W20260808 15:56:00.010782 5449 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:00.001619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:42227 (local address 127.27.254.62:38569)
0808 15:56:00.001767 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:00.001771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:00.002236 (+ 465us) server_negotiation.cc:407] Connection header received
0808 15:56:00.002374 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:00.002378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:00.002434 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:00.002510 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:00.003181 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.003895 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.005064 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.005548 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.005649 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:00.010374 (+ 4725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:42227: BlockingRecv error: recv got EOF from 127.27.254.3:42227 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 15:56:00 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:00 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:00.432376 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:00.423924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:35053)
0808 15:56:00.424463 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:00.424482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:00.424619 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:00.424924 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:00.424927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:00.425311 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:00.425531 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:00.425542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.426502 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.426505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:00.427398 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:00.427409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.428016 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.428020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:00.428142 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:00.428716 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:00.428734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:00.432123 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":321,"threads_started":1}
W20260808 15:56:00.432564 5449 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:00.423975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:35053 (local address 127.27.254.62:38569)
0808 15:56:00.424174 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:00.424181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:00.424574 (+ 393us) server_negotiation.cc:407] Connection header received
0808 15:56:00.424728 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:00.424732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:00.424795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:00.424879 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:00.425666 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.426389 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.427517 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.427895 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.427981 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:00.432324 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:35053: BlockingRecv error: recv got EOF from 127.27.254.2:35053 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 08 15:56:00 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:01 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:01.003521 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:00.994245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:40535)
0808 15:56:00.994893 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:00.994918 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:00.995095 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:00.995465 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:00.995470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:00.995861 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:00.996030 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:00.996038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.997142 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.997148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:00.998094 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:00.998105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.998825 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.998830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:00.999025 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:00.999632 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:00.999650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:01.003179 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
W20260808 15:56:01.003681 5453 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:00.994437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:40535 (local address 127.27.254.62:38569)
0808 15:56:00.995155 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:00.995160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:00.995179 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:56:00.995232 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:00.995236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:00.995317 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:00.995435 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:00.996198 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.996995 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.998251 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:00.998672 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:00.998802 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:01.003408 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:40535: BlockingRecv error: recv got EOF from 127.27.254.1:40535 (error 108)
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":424,"threads_started":1}
Aug 08 15:56:01 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:01 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:01.020128 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:01.011331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:50577)
0808 15:56:01.011844 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:01.011860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:01.011981 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:01.012271 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:01.012274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:01.012644 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:01.012826 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:01.012835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.013832 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.013836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:01.014752 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:01.014765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.015537 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.015542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:01.015680 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:01.016249 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:01.016266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:01.019871 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":303,"threads_started":1}
W20260808 15:56:01.020298 5453 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:01.011354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:50577 (local address 127.27.254.62:38569)
0808 15:56:01.011527 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:01.011531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:01.011934 (+ 403us) server_negotiation.cc:407] Connection header received
0808 15:56:01.012085 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:01.012089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:01.012148 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:01.012239 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:01.012950 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.013697 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.014906 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.015363 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.015469 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:01.020060 (+ 4591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:50577: BlockingRecv error: recv got EOF from 127.27.254.3:50577 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 15:56:01 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:01 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:01.442449 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:01.433461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:45441)
0808 15:56:01.434014 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:01.434030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:01.434177 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:01.434555 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:01.434562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:01.434999 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:01.435255 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:01.435266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.436406 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.436410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:01.437306 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:01.437317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.438037 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.438041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:01.438162 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:01.438707 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:01.438724 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:01.442215 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":321,"threads_started":1}
W20260808 15:56:01.442688 5453 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:01.433524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:45441 (local address 127.27.254.62:38569)
0808 15:56:01.433708 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:01.433713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:01.434130 (+ 417us) server_negotiation.cc:407] Connection header received
0808 15:56:01.434329 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:01.434333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:01.434397 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:01.434489 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:01.435394 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.436283 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.437443 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:01.437915 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:01.438018 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:01.442417 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:45441: BlockingRecv error: recv got EOF from 127.27.254.2:45441 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 15:56:02 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:02 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:02.014384 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:02.004607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:33729)
0808 15:56:02.005235 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:02.005260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:02.005428 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:02.005882 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:02.005889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:02.006508 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:02.006769 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:02.006784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.008122 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.008126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:02.009120 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:02.009131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.009862 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.009866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:02.010007 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:02.010633 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:02.010650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:02.014016 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":308,"threads_started":1}
W20260808 15:56:02.014499 5457 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:02.004662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:33729 (local address 127.27.254.62:38569)
0808 15:56:02.005487 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:02.005494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:02.005519 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:56:02.005594 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:02.005599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:02.005684 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:02.005818 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:02.006940 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.007993 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.009239 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.009739 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.009833 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:02.014240 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:33729: BlockingRecv error: recv got EOF from 127.27.254.1:33729 (error 108)
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":591,"threads_started":1}
Aug 08 15:56:02 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:02 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:02.030011 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:02.021030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:43935)
0808 15:56:02.021529 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:02.021545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:02.021650 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:02.021923 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:02.021926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:02.022263 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:02.022431 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:02.022440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.023391 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.023395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:02.024342 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:02.024357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.025042 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.025047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:02.025172 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:02.025776 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:02.025794 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:02.029640 (+ 3846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
W20260808 15:56:02.030072 5457 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:02.021074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:43935 (local address 127.27.254.62:38569)
0808 15:56:02.021277 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:02.021281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:02.021612 (+ 331us) server_negotiation.cc:407] Connection header received
0808 15:56:02.021741 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:02.021745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:02.021802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:02.021882 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:02.022553 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.023240 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.024475 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.024879 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.024969 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:02.029832 (+ 4863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:43935: BlockingRecv error: recv got EOF from 127.27.254.3:43935 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 08 15:56:02 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:02 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:02.452278 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:02.443408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:41437)
0808 15:56:02.443934 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:02.443950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:02.444062 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:02.444375 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:02.444378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:02.444756 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:02.444907 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:02.444916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.445909 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.445912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:02.446887 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:02.446898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.447591 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.447594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:02.447711 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:02.448328 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:02.448345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:02.452034 (+ 3689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
W20260808 15:56:02.452454 5457 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:02.443525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:41437 (local address 127.27.254.62:38569)
0808 15:56:02.443818 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:02.443825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:02.444051 (+ 226us) server_negotiation.cc:407] Connection header received
0808 15:56:02.444186 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:02.444190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:02.444251 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:02.444331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:02.445033 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.445784 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.447040 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:02.447464 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:02.447568 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:02.452226 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:41437: BlockingRecv error: recv got EOF from 127.27.254.2:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 08 15:56:03 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:03 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:03.024503 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:03.015358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:33607)
0808 15:56:03.015962 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:03.015984 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:03.016118 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:03.016590 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:03.016594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:03.017086 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:03.017237 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:03.017246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.018426 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.018433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:03.019360 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:03.019371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.020047 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.020052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:03.020247 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:03.020857 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:03.020874 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:03.024181 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":327,"threads_started":1}
W20260808 15:56:03.024868 5461 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:03.015421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:33607 (local address 127.27.254.62:38569)
0808 15:56:03.016214 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:03.016222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:03.016239 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:56:03.016290 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:03.016294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:03.016390 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:03.016500 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:03.017385 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.018299 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.019502 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.019905 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.020037 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:03.024398 (+ 4361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:33607: BlockingRecv error: recv got EOF from 127.27.254.1:33607 (error 108)
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":524,"threads_started":1}
Aug 08 15:56:03 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:03 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:03.039258 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:03.030911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:43941)
0808 15:56:03.031438 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:03.031460 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:03.031563 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:03.031845 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:03.031847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:03.032192 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:03.032353 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:03.032362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.033258 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.033262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:03.034314 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:03.034326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.034939 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.034943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:03.035063 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:03.035603 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:03.035619 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:03.038907 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":297,"threads_started":1}
W20260808 15:56:03.039366 5461 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:03.031000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:43941 (local address 127.27.254.62:38569)
0808 15:56:03.031146 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:03.031150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:03.031526 (+ 376us) server_negotiation.cc:407] Connection header received
0808 15:56:03.031657 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:03.031661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:03.031715 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:03.031791 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:03.032465 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.033147 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.034437 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.034822 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.034910 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:03.039103 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:43941: BlockingRecv error: recv got EOF from 127.27.254.3:43941 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 15:56:03 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:03 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:03.467419 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:03.453339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:53737)
0808 15:56:03.453875 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:03.453892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:03.454002 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:03.454275 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:03.454278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:03.454691 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:03.454835 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:03.454843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.455806 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.455810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:03.456770 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:03.456780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.457703 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.457707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:03.457907 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:03.463692 (+ 5785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:03.463710 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:03.467199 (+ 3489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":308,"threads_started":1}
W20260808 15:56:03.467607 5461 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:03.453580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:53737 (local address 127.27.254.62:38569)
0808 15:56:03.453821 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:03.453829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:03.453944 (+ 115us) server_negotiation.cc:407] Connection header received
0808 15:56:03.454088 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:03.454091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:03.454150 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:03.454228 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:03.454967 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.455682 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.456888 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:03.457539 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:03.457686 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:03.467382 (+ 9696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:53737: BlockingRecv error: recv got EOF from 127.27.254.2:53737 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 15:56:04 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:04 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:04.034009 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:04.025386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:49837)
0808 15:56:04.025960 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:04.025978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:04.026084 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:04.026389 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:04.026392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:04.026782 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:04.026961 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:04.026970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.027966 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.027970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:04.028867 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:04.028878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.029571 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.029576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:04.029700 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:04.030337 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:04.030355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:04.033779 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":300,"threads_started":1}
W20260808 15:56:04.034217 5465 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:04.025514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:49837 (local address 127.27.254.62:38569)
0808 15:56:04.026021 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:04.026031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:04.026048 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:56:04.026196 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:04.026200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:04.026259 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:04.026345 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:04.027094 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.027840 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.028984 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.029382 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.029548 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:04.033969 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:49837: BlockingRecv error: recv got EOF from 127.27.254.1:49837 (error 108)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":286,"threads_started":1}
Aug 08 15:56:04 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:04 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:04.049225 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:04.040334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:40867)
0808 15:56:04.040829 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:04.040845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:04.040997 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:04.041333 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:04.041337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:04.041791 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:04.041931 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:04.041939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.042948 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.042954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:04.043860 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:04.043870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.044574 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.044580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:04.044748 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:04.045312 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:04.045329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:04.048997 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":310,"threads_started":1}
W20260808 15:56:04.049538 5465 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:04.040392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:40867 (local address 127.27.254.62:38569)
0808 15:56:04.040577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:04.040585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:04.040929 (+ 344us) server_negotiation.cc:407] Connection header received
0808 15:56:04.041130 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:04.041142 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:04.041200 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:04.041299 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:04.042074 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.042811 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.044004 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.044429 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.044559 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:04.049176 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:40867: BlockingRecv error: recv got EOF from 127.27.254.3:40867 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 15:56:04 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:04 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:04.477476 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:04.468386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:47961)
0808 15:56:04.468868 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:04.468882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:04.469039 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:04.469377 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:04.469381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:04.469814 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:04.469967 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:04.469991 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.471029 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.471040 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:04.472010 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:04.472021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.472830 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.472835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:04.473016 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:04.473642 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:04.473658 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:04.477097 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
W20260808 15:56:04.477632 5465 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:04.468454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:47961 (local address 127.27.254.62:38569)
0808 15:56:04.468701 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:04.468709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:04.468966 (+ 257us) server_negotiation.cc:407] Connection header received
0808 15:56:04.469173 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:04.469177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:04.469247 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:04.469346 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:04.470153 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.470882 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.472198 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:04.472639 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:04.472798 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:04.477332 (+ 4534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:47961: BlockingRecv error: recv got EOF from 127.27.254.2:47961 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 15:56:05 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:05 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:05.043844 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:05.034971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:60681)
0808 15:56:05.035769 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:05.035785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:05.035896 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:05.036210 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:05.036213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:05.036569 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:05.036729 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:05.036736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.037653 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.037656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:05.038509 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:05.038518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.039140 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.039143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:05.039241 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:05.039920 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:05.039936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:05.043607 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":492,"threads_started":1}
W20260808 15:56:05.044116 5468 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:05.035054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:60681 (local address 127.27.254.62:38569)
0808 15:56:05.035500 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:05.035504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:05.035860 (+ 356us) server_negotiation.cc:407] Connection header received
0808 15:56:05.036006 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:05.036009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:05.036069 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:05.036158 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:05.036840 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.037542 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.038624 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.039016 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.039105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:05.043785 (+ 4680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:60681: BlockingRecv error: recv got EOF from 127.27.254.1:60681 (error 108)
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
Aug 08 15:56:05 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:05 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:05.059298 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:05.050103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:59403)
0808 15:56:05.050789 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:05.050810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:05.050923 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:05.051195 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:05.051198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:05.051562 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:05.051741 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:05.051750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.052729 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.052732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:05.054004 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:05.054017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.054948 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.054953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:05.055073 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:05.055701 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:05.055718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:05.059067 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":473,"threads_started":1}
W20260808 15:56:05.059440 5468 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:05.050200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:59403 (local address 127.27.254.62:38569)
0808 15:56:05.050427 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:05.050436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:05.050890 (+ 454us) server_negotiation.cc:407] Connection header received
0808 15:56:05.051014 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:05.051018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:05.051075 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:05.051153 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:05.051874 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.052612 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.054140 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.054799 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.054912 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:05.059245 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:59403: BlockingRecv error: recv got EOF from 127.27.254.3:59403 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 15:56:05 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:05 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:05.487726 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:05.478573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:54843)
0808 15:56:05.479178 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:05.479195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:05.479306 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:05.479623 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:05.479626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:05.480050 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:05.480237 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:05.480245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.481206 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.481210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:05.482209 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:05.482227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.482900 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.482904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:05.483020 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:05.483580 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:05.483598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:05.487392 (+ 3794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":296,"threads_started":1}
W20260808 15:56:05.487948 5468 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:05.478629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:54843 (local address 127.27.254.62:38569)
0808 15:56:05.478836 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:05.478840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:05.479267 (+ 427us) server_negotiation.cc:407] Connection header received
0808 15:56:05.479416 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:05.479420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:05.479478 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:05.479559 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:05.480356 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.481087 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.482344 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:05.482766 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:05.482859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:05.487618 (+ 4759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:54843: BlockingRecv error: recv got EOF from 127.27.254.2:54843 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 08 15:56:06 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:06 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:06.053933 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:06.044780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:32899)
0808 15:56:06.045390 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:06.045453 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:06.045579 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:06.045901 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:06.045904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:06.046302 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:06.046459 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:06.046467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.047436 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.047453 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:06.048397 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:06.048407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.049392 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.049398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:06.049639 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:06.050360 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:06.050377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:06.053681 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":333,"threads_started":1}
W20260808 15:56:06.054162 5473 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:06.044917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:32899 (local address 127.27.254.62:38569)
0808 15:56:06.045469 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:06.045474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:06.045516 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:56:06.045688 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:06.045692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:06.045753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:06.045857 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:06.046581 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.047310 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.048537 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.049209 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.049304 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:06.053889 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:32899: BlockingRecv error: recv got EOF from 127.27.254.1:32899 (error 108)
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":329,"threads_started":1}
Aug 08 15:56:06 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:06 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:06.069278 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:06.060187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:41697)
0808 15:56:06.060904 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:06.060924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:06.061080 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:06.061732 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:06.061735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:06.062308 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:06.062492 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:06.062501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.063452 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.063456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:06.064383 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:06.064394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.065023 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.065027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:06.065151 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:06.065751 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:06.065777 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:06.069038 (+ 3261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":328,"threads_started":1}
W20260808 15:56:06.069460 5473 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:06.060402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:41697 (local address 127.27.254.62:38569)
0808 15:56:06.060620 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:06.060627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:06.061030 (+ 403us) server_negotiation.cc:407] Connection header received
0808 15:56:06.061205 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:06.061212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:06.061310 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:06.061441 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:06.062619 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.063327 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.064497 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.064899 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.064989 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:06.069220 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:41697: BlockingRecv error: recv got EOF from 127.27.254.3:41697 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 15:56:06 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:06 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:06.498353 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:06.488801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:46799)
0808 15:56:06.489396 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:06.489466 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:06.489590 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:06.489877 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:06.489880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:06.490280 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:06.490447 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:06.490456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.491670 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.491675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:06.492647 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:06.492661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.493381 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.493385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:06.493580 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:06.494226 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:06.494250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:06.498109 (+ 3859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":377,"threads_started":1}
W20260808 15:56:06.498531 5473 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:06.488899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:46799 (local address 127.27.254.62:38569)
0808 15:56:06.489245 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:06.489251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:06.489545 (+ 294us) server_negotiation.cc:407] Connection header received
0808 15:56:06.489692 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:06.489696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:06.489755 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:06.489835 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:06.490587 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.491476 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.492790 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:06.493215 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:06.493309 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:06.498306 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:46799: BlockingRecv error: recv got EOF from 127.27.254.2:46799 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Aug 08 15:56:07 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:07 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:07.063265 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:07.054840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:48079)
0808 15:56:07.055413 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:07.055430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:07.055528 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:07.055823 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:07.055827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:07.056232 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:07.056371 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:07.056379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.057316 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.057320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:07.058255 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:07.058265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.058930 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.058934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:07.059079 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:07.059643 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:07.059659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:07.063038 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
W20260808 15:56:07.063469 5477 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:07.054977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:48079 (local address 127.27.254.62:38569)
0808 15:56:07.055466 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:07.055471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:07.055486 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:56:07.055638 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:07.055642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:07.055699 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:07.055780 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:07.056489 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.057195 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.058372 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.058799 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.058895 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:07.063229 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:48079: BlockingRecv error: recv got EOF from 127.27.254.1:48079 (error 108)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
Aug 08 15:56:07 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:07 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:07.079456 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:07.070167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:55709)
0808 15:56:07.070909 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:07.070924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:07.071032 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:07.071319 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:07.071322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:07.071681 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:07.071839 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:07.071847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.072794 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.072797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:07.073884 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:07.073897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.074605 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.074609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:07.074722 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:07.075327 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:07.075343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:07.079222 (+ 3879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":500,"threads_started":1}
W20260808 15:56:07.079615 5477 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:07.070249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:55709 (local address 127.27.254.62:38569)
0808 15:56:07.070490 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:07.070499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:07.071001 (+ 502us) server_negotiation.cc:407] Connection header received
0808 15:56:07.071127 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:07.071131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:07.071189 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:07.071280 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:07.071949 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.072673 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.074025 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.074423 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.074561 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:07.079402 (+ 4841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:55709: BlockingRecv error: recv got EOF from 127.27.254.3:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 15:56:07 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:07 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:07.508693 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:07.499520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:36197)
0808 15:56:07.500046 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:07.500063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:07.500218 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:07.500577 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:07.500582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:07.501010 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:07.501155 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:07.501163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.502275 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.502282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:07.503248 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:07.503259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.503980 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.503986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:07.504170 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:07.504724 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:07.504741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:07.508471 (+ 3730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":279,"threads_started":1}
W20260808 15:56:07.508909 5477 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:07.499565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:36197 (local address 127.27.254.62:38569)
0808 15:56:07.499788 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:07.499795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:07.500151 (+ 356us) server_negotiation.cc:407] Connection header received
0808 15:56:07.500358 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:07.500362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:07.500433 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:07.500538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:07.501299 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.502109 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.503388 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:07.503828 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:07.503964 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:07.508644 (+ 4680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:36197: BlockingRecv error: recv got EOF from 127.27.254.2:36197 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 15:56:08 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:08 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:08.073662 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:08.064377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:45337)
0808 15:56:08.064950 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:08.064970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:08.065117 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:08.065451 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:08.065454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:08.065795 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:08.066009 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:08.066055 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.067055 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.067060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:08.068016 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:08.068035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.068664 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.068669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:08.068774 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:08.069348 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:08.069371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:08.073313 (+ 3942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":297,"threads_started":1}
W20260808 15:56:08.073863 5481 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:08.064425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:45337 (local address 127.27.254.62:38569)
0808 15:56:08.064892 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:08.064896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:08.065081 (+ 185us) server_negotiation.cc:407] Connection header received
0808 15:56:08.065214 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:08.065217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:08.065288 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:08.065387 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:08.066180 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.066895 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.068142 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.068538 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.068630 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:08.073622 (+ 4992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:45337: BlockingRecv error: recv got EOF from 127.27.254.1:45337 (error 108)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":271,"threads_started":1}
Aug 08 15:56:08 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:08 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:08.089547 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:08.080405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:59781)
0808 15:56:08.080937 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:08.080952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:08.081063 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:08.081485 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:08.081490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:08.081969 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:08.082202 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:08.082220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.083537 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.083541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:08.084464 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:08.084475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.085093 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.085097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:08.085209 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:08.085819 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:08.085836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:08.089264 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":292,"threads_started":1}
W20260808 15:56:08.089779 5481 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:08.080473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:59781 (local address 127.27.254.62:38569)
0808 15:56:08.080630 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:08.080634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:08.081020 (+ 386us) server_negotiation.cc:407] Connection header received
0808 15:56:08.081195 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:08.081200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:08.081276 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:08.081389 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:08.082367 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.083407 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.084591 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.084965 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.085058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:08.089541 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:59781: BlockingRecv error: recv got EOF from 127.27.254.3:59781 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 15:56:08 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:08 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:08.522643 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:08.509635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:41251)
0808 15:56:08.510150 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:08.510166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:08.510286 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:08.510603 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:08.510606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:08.510947 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:08.511135 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:08.511144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.512101 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.512105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:08.513101 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:08.513112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.514087 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.514095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:08.514232 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:08.519012 (+ 4780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:08.519043 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:08.522404 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":280,"threads_started":1}
W20260808 15:56:08.522818 5481 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:08.509794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:41251 (local address 127.27.254.62:38569)
0808 15:56:08.510046 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:08.510053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:08.510247 (+ 194us) server_negotiation.cc:407] Connection header received
0808 15:56:08.510409 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:08.510412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:08.510475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:08.510558 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:08.511261 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.511962 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.513232 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:08.513876 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:08.514053 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:08.522595 (+ 8542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:41251: BlockingRecv error: recv got EOF from 127.27.254.2:41251 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 08 15:56:09 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:09 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:09.083740 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:09.074619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:46539)
0808 15:56:09.075150 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:09.075167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:09.075270 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:09.075677 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:09.075681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:09.076086 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:09.076277 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:09.076286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.077435 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.077442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:09.078486 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:09.078499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.079310 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.079316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:09.079492 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:09.080066 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:09.080082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:09.083508 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":292,"threads_started":1}
W20260808 15:56:09.083969 5485 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:09.074798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:46539 (local address 127.27.254.62:38569)
0808 15:56:09.075356 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:09.075362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:09.075383 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:56:09.075440 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:09.075444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:09.075500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:09.075597 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:09.076380 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.077261 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.078648 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.079150 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.079289 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:09.083722 (+ 4433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:46539: BlockingRecv error: recv got EOF from 127.27.254.1:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":299,"threads_started":1}
Aug 08 15:56:09 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:09 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:09.099056 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:09.090527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:60613)
0808 15:56:09.091264 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:09.091283 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:09.091411 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:09.091707 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:09.091712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:09.092113 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:09.092266 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:09.092275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.093201 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.093205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:09.094211 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:09.094222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.094863 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.094867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:09.094972 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:09.095494 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:09.095510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:09.098795 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":505,"threads_started":1}
W20260808 15:56:09.099146 5485 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:09.090672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:60613 (local address 127.27.254.62:38569)
0808 15:56:09.090860 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:09.090865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:09.091370 (+ 505us) server_negotiation.cc:407] Connection header received
0808 15:56:09.091514 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:09.091517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:09.091576 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:09.091650 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:09.092385 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.093093 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.094322 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.094730 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.094818 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:09.098958 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:60613: BlockingRecv error: recv got EOF from 127.27.254.3:60613 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 08 15:56:09 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:09 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:09.532491 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:09.523673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:56171)
0808 15:56:09.524170 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:09.524186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:09.524298 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:09.524599 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:09.524603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:09.525003 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:09.525199 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:09.525208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.526295 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.526299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:09.527196 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:09.527207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.527889 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.527894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:09.528045 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:09.528653 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:09.528670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:09.532155 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":282,"threads_started":1}
W20260808 15:56:09.532612 5485 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:09.523760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:56171 (local address 127.27.254.62:38569)
0808 15:56:09.523981 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:09.523990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:09.524265 (+ 275us) server_negotiation.cc:407] Connection header received
0808 15:56:09.524401 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:09.524405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:09.524475 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:09.524554 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:09.525316 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.526167 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.527336 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:09.527734 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:09.527825 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:09.532328 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:56171: BlockingRecv error: recv got EOF from 127.27.254.2:56171 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 15:56:10 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:10 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:10.094069 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:10.084691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:48671)
0808 15:56:10.085284 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:10.085300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:10.085478 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:10.085839 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:10.085842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:10.086272 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:10.086436 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:10.086445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.087633 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.087645 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:10.088587 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:10.088598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.089391 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.089398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:10.089598 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:10.090239 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:10.090257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:10.093827 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":343,"threads_started":1}
W20260808 15:56:10.094429 5489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:10.084820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:48671 (local address 127.27.254.62:38569)
0808 15:56:10.085302 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:10.085306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:10.085378 (+ 72us) server_negotiation.cc:407] Connection header received
0808 15:56:10.085622 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:10.085626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:10.085695 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:10.085799 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:10.086592 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.087473 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.088725 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.089215 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.089355 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:10.094049 (+ 4694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:48671: BlockingRecv error: recv got EOF from 127.27.254.1:48671 (error 108)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
Aug 08 15:56:10 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:10 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:10.108649 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:10.100032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:33571)
0808 15:56:10.100516 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:10.100535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:10.100640 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:10.101032 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:10.101035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:10.101525 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:10.101716 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:10.101724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.102693 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.102697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:10.103608 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:10.103620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.104283 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.104287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:10.104409 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:10.104938 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:10.104955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:10.108327 (+ 3372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":271,"threads_started":1}
W20260808 15:56:10.108727 5489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:10.100089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:33571 (local address 127.27.254.62:38569)
0808 15:56:10.100238 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:10.100243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:10.100605 (+ 362us) server_negotiation.cc:407] Connection header received
0808 15:56:10.100752 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:10.100757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:10.100831 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:10.100928 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:10.101842 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.102570 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.103742 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.104152 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.104246 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:10.108494 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:33571: BlockingRecv error: recv got EOF from 127.27.254.3:33571 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 15:56:10 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:10 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:10.542546 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:10.533579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:49199)
0808 15:56:10.534084 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:10.534099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:10.534211 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:10.534557 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:10.534560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:10.534973 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:10.535165 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:10.535174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.536129 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.536133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:10.537038 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:10.537049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.537807 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.537811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:10.537945 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:10.538686 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:10.538710 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:10.542298 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
W20260808 15:56:10.542768 5489 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:10.533616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:49199 (local address 127.27.254.62:38569)
0808 15:56:10.533845 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:10.533854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:10.534182 (+ 328us) server_negotiation.cc:407] Connection header received
0808 15:56:10.534313 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:10.534317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:10.534392 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:10.534471 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:10.535293 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.536008 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.537160 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:10.537644 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:10.537744 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:10.542485 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:49199: BlockingRecv error: recv got EOF from 127.27.254.2:49199 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 15:56:11 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:11 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:11.103832 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:11.095001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:60885)
0808 15:56:11.095529 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:11.095545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:11.095642 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:11.095933 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:11.095936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:11.096327 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:11.096478 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:11.096486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.097495 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.097499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:11.098398 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:11.098408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.099039 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.099043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:11.099162 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:11.099705 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:11.099721 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:11.103494 (+ 3773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":315,"threads_started":1}
W20260808 15:56:11.104197 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:11.095040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:60885 (local address 127.27.254.62:38569)
0808 15:56:11.095606 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:11.095610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:11.095626 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:56:11.095744 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:11.095747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:11.095806 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:11.095893 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:11.096601 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.097313 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.098518 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.098924 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.099023 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:11.103699 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:60885: BlockingRecv error: recv got EOF from 127.27.254.1:60885 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":385,"threads_started":1}
Aug 08 15:56:11 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:11 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:11.118532 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:11.109749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:40367)
0808 15:56:11.110247 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:11.110262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:11.110365 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:11.110631 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:11.110634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:11.111000 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:11.111156 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:11.111164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.111991 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.111994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:11.112880 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:11.112892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.113678 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.113682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:11.113806 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:11.114431 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:11.114448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:11.118300 (+ 3852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
W20260808 15:56:11.118718 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:11.109790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:40367 (local address 127.27.254.62:38569)
0808 15:56:11.110042 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:11.110048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:11.110337 (+ 289us) server_negotiation.cc:407] Connection header received
0808 15:56:11.110452 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:11.110455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:11.110512 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:11.110585 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:11.111291 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.111877 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.113028 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.113523 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.113619 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:11.118500 (+ 4881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:40367: BlockingRecv error: recv got EOF from 127.27.254.3:40367 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 15:56:11 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:11 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:11.552500 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:11.543518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:37161)
0808 15:56:11.544042 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:11.544058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:11.544168 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:11.544473 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:11.544476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:11.544864 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:11.545006 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:11.545014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.546349 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.546353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:11.547553 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:11.547574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.548204 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.548208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:11.548326 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:11.548892 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:11.548909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:11.552256 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":301,"threads_started":1}
W20260808 15:56:11.552734 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:11.543699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:37161 (local address 127.27.254.62:38569)
0808 15:56:11.543872 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:11.543876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:11.544129 (+ 253us) server_negotiation.cc:407] Connection header received
0808 15:56:11.544286 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:11.544289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:11.544348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:11.544431 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:11.545124 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.546222 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.547692 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:11.548086 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:11.548178 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:11.552455 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:37161: BlockingRecv error: recv got EOF from 127.27.254.2:37161 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 15:56:12 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:12 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.1@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:12.113903 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:12.104970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.1:58083)
0808 15:56:12.105507 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:12.105524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:12.105622 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:12.105923 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:12.105926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:12.106349 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:12.106584 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:12.106596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.107761 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.107766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:12.108860 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:12.108873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.109641 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.109645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:12.109770 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.110306 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:12.110322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:12.113657 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
W20260808 15:56:12.114132 5497 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:12.105065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:58083 (local address 127.27.254.62:38569)
0808 15:56:12.105539 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:12.105543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:12.105571 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:56:12.105726 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:12.105730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:12.105794 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:12.105881 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:12.106736 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.107597 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.108984 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.109516 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.109616 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.113845 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.1:58083: BlockingRecv error: recv got EOF from 127.27.254.1:58083 (error 108)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
Aug 08 15:56:12 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:12 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.3@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:12.127898 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:12.119398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.3:37651)
0808 15:56:12.119870 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:12.119901 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:12.120029 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:12.120406 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:12.120411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:12.120896 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:12.121040 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:12.121049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.122148 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.122172 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:12.123073 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:12.123084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.123775 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.123781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:12.123948 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.124452 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:12.124469 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:12.127588 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
W20260808 15:56:12.127950 5497 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:12.119456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:37651 (local address 127.27.254.62:38569)
0808 15:56:12.119813 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:12.119819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:12.119982 (+ 163us) server_negotiation.cc:407] Connection header received
0808 15:56:12.120189 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:12.120194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:12.120269 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:12.120379 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:12.121177 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.122006 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.123222 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.123629 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.123754 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.127748 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.3:37651: BlockingRecv error: recv got EOF from 127.27.254.3:37651 (error 108)
Metrics: {"server-negotiator.queue_time_us":250}
Aug 08 15:56:12 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:56:12 dist-test-slave-srgv krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.254.2@KRBTEST.COM for oryx/127.27.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:56:12.562335 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:12.553738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38569 (local address 127.27.254.2:52315)
0808 15:56:12.554277 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:12.554294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:12.554414 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:12.554711 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:12.554714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:12.555061 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:12.555234 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:12.555242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.556199 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.556202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:12.557112 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:12.557129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.557886 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.557893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:12.558030 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.558803 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:12.558820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:12.562084 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.254.62:38569: Server oryx/127.27.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":314,"threads_started":1}
W20260808 15:56:12.562518 5497 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:12.553846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:52315 (local address 127.27.254.62:38569)
0808 15:56:12.554010 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:12.554014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:12.554376 (+ 362us) server_negotiation.cc:407] Connection header received
0808 15:56:12.554509 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:12.554512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:12.554581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:12.554663 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:12.555346 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.556076 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.557255 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.557741 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.557832 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.562281 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.254.2:52315: BlockingRecv error: recv got EOF from 127.27.254.2:52315 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:56:12.856742 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5023
I20260808 15:56:12.869340 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:12.857947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.27.254.1:39165)
0808 15:56:12.858648 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:12.858656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:12.858680 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:56:12.858743 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:12.858750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:12.858978 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:12.859150 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:12.860565 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.861661 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.862877 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.863418 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.863522 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.866069 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:12.866077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:12.866091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:12.866143 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:12.868822 (+ 2679us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.254.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:56:12.869112 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50628: Request ticket server kudu/127.27.254.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":368,"threads_started":1}
W20260808 15:56:12.869481 5501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50628: Request ticket server kudu/127.27.254.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:56:12.869520 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5023: Not authorized: Client connection negotiation failed: client connection to 127.27.254.1:39165: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.254.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:56:12.869789 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5023
I20260808 15:56:12.886605 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5175
I20260808 15:56:12.899515 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:12.887804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.27.254.2:34471)
0808 15:56:12.888522 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:12.888531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:12.888568 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:56:12.888652 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:12.888659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:12.888906 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:12.889115 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:12.890519 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.891600 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.892844 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.893497 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.893643 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.896132 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:12.896142 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:12.896156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:12.896213 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:12.898775 (+ 2562us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.254.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:56:12.899170 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37078: Request ticket server kudu/127.27.254.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
W20260808 15:56:12.899560 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5175: Not authorized: Client connection negotiation failed: client connection to 127.27.254.2:34471: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.254.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:56:12.899639 5502 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37078: Request ticket server kudu/127.27.254.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:56:12.899722 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5175
I20260808 15:56:12.916683 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5308
I20260808 15:56:12.929375 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:12.917886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.27.254.3:43565)
0808 15:56:12.918404 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:12.918411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:12.918436 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:56:12.918508 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:12.918515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:12.918743 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:12.918967 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:12.920409 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.921470 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.922709 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:12.923305 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:12.923448 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:12.926108 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:12.926116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:12.926131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:12.926177 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:12.928664 (+ 2487us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.254.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:56:12.929024 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47486: Request ticket server kudu/127.27.254.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
W20260808 15:56:12.929510 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5308: Not authorized: Client connection negotiation failed: client connection to 127.27.254.3:43565: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.254.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:56:12.929615 5503 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47486: Request ticket server kudu/127.27.254.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:56:12.929682 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5308
I20260808 15:56:12.947535 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 4954
I20260808 15:56:12.948882 5016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:13.067852 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 4954
2026-08-08T15:56:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21370 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T15:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:56:13Z Disabled control of system clock
I20260808 15:56:13.144641 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:39151
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 15:56:13.470484 5508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:13.470759 5508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:13.470899 5508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:13.478070 5508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:56:13.478161 5508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:13.478271 5508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:13.478372 5508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:56:13.478446 5508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:56:13.487423 5508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:39151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:13.488741 5508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:13.490531 5508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:13.501854 5513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:13.501894 5514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:13.501920 5516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:13.502691 5508 server_base.cc:1034] running on GCE node
I20260808 15:56:13.503396 5508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:13.504720 5508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:13.505946 5508 hybrid_clock.cc:648] HybridClock initialized: now 1786204573505895 us; error 52 us; skew 500 ppm
I20260808 15:56:13.506270 5508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:13.508351 5508 webserver.cc:460] Webserver started at http://127.27.254.62:32771/ using document root <none> and password file <none>
I20260808 15:56:13.509053 5508 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:13.509181 5508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:13.509567 5508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:13.512372 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "2f22a973c33c4328a81415ed0b98adb6"
format_stamp: "Formatted at 2026-08-08 15:56:13 on dist-test-slave-srgv"
server_key: "889c5df7fe14ef995e94b4703907268b"
server_key_iv: "185a63619e178cdb8cb8f7f9492928e7"
server_key_version: "encryptionkey@0"
I20260808 15:56:13.513164 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "2f22a973c33c4328a81415ed0b98adb6"
format_stamp: "Formatted at 2026-08-08 15:56:13 on dist-test-slave-srgv"
server_key: "889c5df7fe14ef995e94b4703907268b"
server_key_iv: "185a63619e178cdb8cb8f7f9492928e7"
server_key_version: "encryptionkey@0"
I20260808 15:56:13.518201 5508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260808 15:56:13.521486 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:13.522749 5508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:56:13.522924 5508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "2f22a973c33c4328a81415ed0b98adb6"
format_stamp: "Formatted at 2026-08-08 15:56:13 on dist-test-slave-srgv"
server_key: "889c5df7fe14ef995e94b4703907268b"
server_key_iv: "185a63619e178cdb8cb8f7f9492928e7"
server_key_version: "encryptionkey@0"
I20260808 15:56:13.523145 5508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:13.548266 5508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:13.549118 5508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:13.549422 5508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:13.567427 5508 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:39151
I20260808 15:56:13.567451 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:39151 every 8 connection(s)
I20260808 15:56:13.568747 5508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:56:13.572403 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:56:13.576215 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5508
I20260808 15:56:13.576435 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:56:13.577258 28664 external_mini_cluster.cc:1428] Setting key a2b677ddd43ec5b374be9e5a132d0ca1
I20260808 15:56:13.584113 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6: Bootstrap starting.
I20260808 15:56:13.587601 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6: Neither blocks nor log segments found. Creating new log.
I20260808 15:56:13.588814 5574 log.cc:824] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6: Log is configured to *not* fsync() on all Append() calls
I20260808 15:56:13.591572 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:13.579370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.27.254.62:39151)
0808 15:56:13.580334 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:13.580344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:13.580382 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:56:13.582645 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:13.582667 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:13.583105 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:13.583409 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:13.584079 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:13.585380 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:13.587254 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:13.588007 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:13.588142 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:13.589300 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:13.589307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:13.589319 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:13.589923 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:13.590030 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:13.590051 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:13.590185 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:13.590408 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 15:56:13.590700 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":355,"threads_started":1}
I20260808 15:56:13.591845 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6: No bootstrap required, opened a new log
I20260808 15:56:13.599578 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f22a973c33c4328a81415ed0b98adb6" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39151 } }
I20260808 15:56:13.599944 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:56:13.600265 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f22a973c33c4328a81415ed0b98adb6, State: Initialized, Role: FOLLOWER
I20260808 15:56:13.601006 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f22a973c33c4328a81415ed0b98adb6" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39151 } }
I20260808 15:56:13.601265 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:56:13.601475 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:56:13.601662 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:56:13.604218 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f22a973c33c4328a81415ed0b98adb6" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39151 } }
I20260808 15:56:13.604727 5574 leader_election.cc:302] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f22a973c33c4328a81415ed0b98adb6; no voters:
I20260808 15:56:13.605324 5574 leader_election.cc:288] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:56:13.605544 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:56:13.606262 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [term 1 LEADER]: Becoming Leader. State: Replica: 2f22a973c33c4328a81415ed0b98adb6, State: Running, Role: LEADER
I20260808 15:56:13.606878 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f22a973c33c4328a81415ed0b98adb6" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39151 } }
I20260808 15:56:13.607455 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:56:13.609503 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f22a973c33c4328a81415ed0b98adb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f22a973c33c4328a81415ed0b98adb6" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39151 } } }
I20260808 15:56:13.609876 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:56:13.610491 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f22a973c33c4328a81415ed0b98adb6. Latest consensus state: current_term: 1 leader_uuid: "2f22a973c33c4328a81415ed0b98adb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f22a973c33c4328a81415ed0b98adb6" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39151 } } }
I20260808 15:56:13.610869 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:56:13.611632 5588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:56:13.616941 5588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:56:13.624912 5588 catalog_manager.cc:1347] Generated new cluster ID: d51938fbcb654d078b6a8249d8e8fe2c
I20260808 15:56:13.625027 5588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:56:13.637861 5588 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:56:13.639653 5588 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:56:13.648597 5588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f22a973c33c4328a81415ed0b98adb6: Generated new TSK 0
I20260808 15:56:13.649634 5588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:56:13.666376 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39151
--builtin_ntp_servers=127.27.254.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 15:56:14.001547 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:14.001878 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:14.002034 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:14.009446 5599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:14.009584 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:14.009752 5599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:56:14.018951 5599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:14.020287 5599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:14.022087 5599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:14.035370 5605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:14.035390 5607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:14.035966 5604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:14.036464 5599 server_base.cc:1034] running on GCE node
I20260808 15:56:14.037057 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:14.038065 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:14.039274 5599 hybrid_clock.cc:648] HybridClock initialized: now 1786204574039238 us; error 54 us; skew 500 ppm
I20260808 15:56:14.039597 5599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:14.041910 5599 webserver.cc:460] Webserver started at http://127.27.254.1:34063/ using document root <none> and password file <none>
I20260808 15:56:14.042624 5599 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:14.042764 5599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:14.043089 5599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:14.045967 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "aecce5dc557d405abe28ddce5ffa9a88"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "98923fec60a0622a481ebb6301a17f35"
server_key_iv: "7276aa6cb5b6581fee6cb6ee6bfe12cd"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.046761 5599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "aecce5dc557d405abe28ddce5ffa9a88"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "98923fec60a0622a481ebb6301a17f35"
server_key_iv: "7276aa6cb5b6581fee6cb6ee6bfe12cd"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.051780 5599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:56:14.055181 5613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.056458 5599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:56:14.056644 5599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "aecce5dc557d405abe28ddce5ffa9a88"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "98923fec60a0622a481ebb6301a17f35"
server_key_iv: "7276aa6cb5b6581fee6cb6ee6bfe12cd"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.056877 5599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:14.091567 5599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:14.092445 5599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:14.092746 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:14.093818 5599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:14.095224 5599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:14.095347 5599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.095477 5599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:14.095572 5599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.132486 5599 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:42709
I20260808 15:56:14.132581 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:42709 every 8 connection(s)
I20260808 15:56:14.133854 5599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:56:14.141785 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5599
I20260808 15:56:14.141983 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:56:14.142652 28664 external_mini_cluster.cc:1428] Setting key b2b815c64a8a4800623491492b8b551f
I20260808 15:56:14.145393 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39151
--builtin_ntp_servers=127.27.254.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 15:56:14.146502 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:14.136345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:37971 (local address 127.27.254.62:39151)
0808 15:56:14.136796 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:14.136801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:14.137969 (+ 1168us) server_negotiation.cc:407] Connection header received
0808 15:56:14.139724 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:14.139729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:14.139816 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:14.139950 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:14.141022 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.141874 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.143690 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.144136 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.144277 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:14.145638 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:14.145644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:14.145649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:14.145676 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:14.145717 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:14.145729 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:14.145826 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:14.146055 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 15:56:14.146160 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":244,"threads_started":1}
I20260808 15:56:14.147490 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:14.136900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:39151 (local address 127.27.254.1:37971)
0808 15:56:14.137814 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:14.137848 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:14.139442 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:14.140183 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:14.140192 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:14.140279 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:14.140850 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:14.140866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.142034 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.142041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:14.143532 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:14.143545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.144361 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.144368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:14.144605 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:14.145307 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:56:14.145310 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:14.145347 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:56:14.145445 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:14.145825 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:14.145837 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:14.145848 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:56:14.146045 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 15:56:14.146432 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":247,"threads_started":1}
I20260808 15:56:14.149296 5726 heartbeater.cc:347] Connected to a master server at 127.27.254.62:39151
I20260808 15:56:14.149614 5726 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:14.150360 5726 heartbeater.cc:511] Master 127.27.254.62:39151 requested a full tablet report, sending...
I20260808 15:56:14.152602 5539 ts_manager.cc:194] Registered new tserver with Master: aecce5dc557d405abe28ddce5ffa9a88 (127.27.254.1:42709)
I20260808 15:56:14.154249 5539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.1:37971
W20260808 15:56:14.476995 5730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:14.477272 5730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:14.477360 5730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:14.485486 5730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:14.485597 5730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:14.485704 5730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:56:14.494958 5730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:14.496186 5730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:14.497952 5730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:14.511127 5738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:14.511227 5735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:14.511214 5736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:14.514803 5730 server_base.cc:1034] running on GCE node
I20260808 15:56:14.515416 5730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:14.516345 5730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:14.517553 5730 hybrid_clock.cc:648] HybridClock initialized: now 1786204574517512 us; error 56 us; skew 500 ppm
I20260808 15:56:14.517881 5730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:14.520133 5730 webserver.cc:460] Webserver started at http://127.27.254.2:36781/ using document root <none> and password file <none>
I20260808 15:56:14.520833 5730 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:14.520965 5730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:14.521294 5730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:14.524243 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "51316bb739bb401fac062d01f0aa56bc"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "44d4b0441d6d83f73df2ac59bfa81bed"
server_key_iv: "edbe5ffa4bd14b270ecb80d5cb6dbfbe"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.525029 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "51316bb739bb401fac062d01f0aa56bc"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "44d4b0441d6d83f73df2ac59bfa81bed"
server_key_iv: "edbe5ffa4bd14b270ecb80d5cb6dbfbe"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.530210 5730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:56:14.533674 5744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.534962 5730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:56:14.535249 5730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "51316bb739bb401fac062d01f0aa56bc"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "44d4b0441d6d83f73df2ac59bfa81bed"
server_key_iv: "edbe5ffa4bd14b270ecb80d5cb6dbfbe"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.535480 5730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:14.548457 5730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:14.549279 5730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:14.549652 5730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:14.550549 5730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:14.551949 5730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:14.552078 5730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.552202 5730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:14.552289 5730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.587009 5730 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:38543
I20260808 15:56:14.587054 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:38543 every 8 connection(s)
I20260808 15:56:14.588313 5730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:56:14.591473 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5730
I20260808 15:56:14.591655 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:56:14.592126 28664 external_mini_cluster.cc:1428] Setting key 6efe9a6e3747a9dd17d88673958231c7
I20260808 15:56:14.595145 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39151
--builtin_ntp_servers=127.27.254.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 15:56:14.601846 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:14.590867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:35475 (local address 127.27.254.62:39151)
0808 15:56:14.591044 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:14.591049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:14.592570 (+ 1521us) server_negotiation.cc:407] Connection header received
0808 15:56:14.594595 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:14.594601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:14.594681 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:14.594791 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:14.595898 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.597102 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.598744 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.599444 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.599578 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:14.600906 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:14.600915 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:14.600922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:14.600948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:14.600992 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:14.601011 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:14.601116 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:14.601303 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 15:56:14.601468 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:56:14.602842 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:14.591343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:39151 (local address 127.27.254.2:35475)
0808 15:56:14.592381 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:14.592425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:14.594328 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:14.594988 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:14.594996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:14.595082 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:14.595729 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:14.595745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.597255 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.597259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:14.598599 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:14.598611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:14.599646 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:14.599650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:14.599852 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:14.600570 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:56:14.600572 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:14.600611 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:56:14.600685 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:14.601109 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:14.601116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:14.601122 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 15:56:14.601262 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 15:56:14.601634 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":343,"threads_started":1}
I20260808 15:56:14.604272 5857 heartbeater.cc:347] Connected to a master server at 127.27.254.62:39151
I20260808 15:56:14.604578 5857 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:14.605322 5857 heartbeater.cc:511] Master 127.27.254.62:39151 requested a full tablet report, sending...
I20260808 15:56:14.606729 5539 ts_manager.cc:194] Registered new tserver with Master: 51316bb739bb401fac062d01f0aa56bc (127.27.254.2:38543)
I20260808 15:56:14.607616 5539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.2:35475
W20260808 15:56:14.926470 5860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:14.926831 5860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:14.926997 5860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:14.934305 5860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:14.934433 5860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:14.934612 5860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:56:14.943969 5860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:14.945314 5860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:14.947093 5860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:14.960065 5865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:14.960361 5866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:14.960291 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:14.961241 5860 server_base.cc:1034] running on GCE node
I20260808 15:56:14.961978 5860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:14.962884 5860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:14.964089 5860 hybrid_clock.cc:648] HybridClock initialized: now 1786204574964053 us; error 33 us; skew 500 ppm
I20260808 15:56:14.964442 5860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:14.966691 5860 webserver.cc:460] Webserver started at http://127.27.254.3:40873/ using document root <none> and password file <none>
I20260808 15:56:14.967401 5860 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:14.967518 5860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:14.967840 5860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:14.970650 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "6bf815be4d7b4274889f07fc2b415bc5"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "4f2f227be43d898b1a0e9fc1c674f947"
server_key_iv: "4f9c00d257360cbe79db5acf691a09f2"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.971416 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "6bf815be4d7b4274889f07fc2b415bc5"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "4f2f227be43d898b1a0e9fc1c674f947"
server_key_iv: "4f9c00d257360cbe79db5acf691a09f2"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.976975 5860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:56:14.981143 5874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.982553 5860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:56:14.982728 5860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "6bf815be4d7b4274889f07fc2b415bc5"
format_stamp: "Formatted at 2026-08-08 15:56:14 on dist-test-slave-srgv"
server_key: "4f2f227be43d898b1a0e9fc1c674f947"
server_key_iv: "4f9c00d257360cbe79db5acf691a09f2"
server_key_version: "encryptionkey@0"
I20260808 15:56:14.982967 5860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:14.995270 5860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:14.996050 5860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:14.996342 5860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:14.997287 5860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:14.998682 5860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:14.998768 5860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:14.998914 5860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:14.998987 5860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:15.031585 5860 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:36391
I20260808 15:56:15.031623 5986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:36391 every 8 connection(s)
I20260808 15:56:15.032883 5860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:56:15.040019 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 5860
I20260808 15:56:15.040206 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:56:15.040644 28664 external_mini_cluster.cc:1428] Setting key 65050851ce17a3a13024b5ebec5ed36d
I20260808 15:56:15.045516 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:15.035448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:36659 (local address 127.27.254.62:39151)
0808 15:56:15.035664 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:15.035670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:15.036963 (+ 1293us) server_negotiation.cc:407] Connection header received
0808 15:56:15.038779 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:15.038784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:15.038862 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:15.038949 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:15.039989 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.040914 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.042825 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.043276 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.043432 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:15.044667 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:15.044674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:15.044678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:15.044703 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:15.044732 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:15.044744 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:15.044838 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:15.045062 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 15:56:15.045158 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:56:15.046504 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:15.035981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:39151 (local address 127.27.254.3:36659)
0808 15:56:15.036811 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:15.036850 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:15.038522 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:15.039160 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:15.039168 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:15.039253 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:15.039802 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:15.039818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.041074 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.041079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:15.042688 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:15.042699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.043520 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.043529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:15.043775 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:15.044360 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:56:15.044362 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:15.044404 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:56:15.044475 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:15.044839 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:15.044852 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:15.044861 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:56:15.045068 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 15:56:15.045470 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":245,"threads_started":1}
I20260808 15:56:15.047750 5987 heartbeater.cc:347] Connected to a master server at 127.27.254.62:39151
I20260808 15:56:15.048051 5987 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:15.048739 5987 heartbeater.cc:511] Master 127.27.254.62:39151 requested a full tablet report, sending...
I20260808 15:56:15.050096 5539 ts_manager.cc:194] Registered new tserver with Master: 6bf815be4d7b4274889f07fc2b415bc5 (127.27.254.3:36391)
I20260808 15:56:15.050868 5539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.3:36659
I20260808 15:56:15.055704 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:56:15.061728 5728 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:15.060217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.27.254.62:39151)
0808 15:56:15.060467 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:15.060472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:15.060872 (+ 400us) server_negotiation.cc:407] Connection header received
0808 15:56:15.061003 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:15.061007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:15.061076 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:15.061162 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:15.061565 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53300: BlockingRecv error: recv got EOF from 127.0.0.1:53300 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:56:15.062130 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5599
I20260808 15:56:15.069319 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:15.063036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.27.254.1:42709)
0808 15:56:15.063570 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:15.063576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:15.063601 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:56:15.063659 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:15.063665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:15.063844 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:15.063967 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:15.064294 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.065261 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.066605 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.067129 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.067223 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:15.068106 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:15.068113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:15.068125 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:15.068668 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:15.068711 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:15.068729 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:15.068814 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:15.068921 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 15:56:15.069022 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":273,"threads_started":1}
I20260808 15:56:15.070497 5721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:15.153703 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5599
I20260808 15:56:15.169886 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5730
I20260808 15:56:15.177675 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:15.170715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.27.254.2:38543)
0808 15:56:15.171229 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:15.171236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:15.171261 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:56:15.171325 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:15.171332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:15.171533 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:15.171721 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:15.172161 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.173141 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.174508 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.175078 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.175223 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:15.176237 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:15.176244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:15.176257 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:15.176827 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:15.176871 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:15.176894 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:15.177017 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:15.177177 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 15:56:15.177321 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260808 15:56:15.178814 5852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:15.259085 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5730
I20260808 15:56:15.275915 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5860
I20260808 15:56:15.284039 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:15.276863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.27.254.3:36391)
0808 15:56:15.277534 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:15.277541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:15.277580 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:56:15.277638 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:15.277644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:15.277856 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:15.278033 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:15.278506 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.279511 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.280778 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:15.281329 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:15.281520 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:15.282613 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:15.282622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:15.282635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:15.283204 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:15.283248 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:15.283266 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:15.283398 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:15.283547 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 15:56:15.283700 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":319,"threads_started":1}
I20260808 15:56:15.285297 5982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:15.366923 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5860
I20260808 15:56:15.384359 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 5508
I20260808 15:56:15.385603 5569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:15.487028 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 5508
2026-08-08T15:56:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2414 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T15:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:56:15Z Disabled control of system clock
I20260808 15:56:15.559437 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33745
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 15:56:15.883633 6006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:15.883906 6006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:15.884076 6006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:15.891279 6006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:56:15.891383 6006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:15.891479 6006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:15.891597 6006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:56:15.891677 6006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:56:15.900769 6006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:15.902079 6006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:15.903865 6006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:15.915509 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:15.915556 6014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:15.915643 6012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:15.916222 6006 server_base.cc:1034] running on GCE node
I20260808 15:56:15.916986 6006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:15.918366 6006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:15.919593 6006 hybrid_clock.cc:648] HybridClock initialized: now 1786204575919567 us; error 45 us; skew 500 ppm
I20260808 15:56:15.919925 6006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:15.922216 6006 webserver.cc:460] Webserver started at http://127.27.254.62:46227/ using document root <none> and password file <none>
I20260808 15:56:15.922896 6006 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:15.923027 6006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:15.923377 6006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:15.926281 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "2a2f6da1739949be8dd9f34612f60d8b"
format_stamp: "Formatted at 2026-08-08 15:56:15 on dist-test-slave-srgv"
server_key: "80ff54e1c6b1550c9d5f76abca9f4d94"
server_key_iv: "deff627c61a95ff2c5fc938c73ea1581"
server_key_version: "encryptionkey@0"
I20260808 15:56:15.927000 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "2a2f6da1739949be8dd9f34612f60d8b"
format_stamp: "Formatted at 2026-08-08 15:56:15 on dist-test-slave-srgv"
server_key: "80ff54e1c6b1550c9d5f76abca9f4d94"
server_key_iv: "deff627c61a95ff2c5fc938c73ea1581"
server_key_version: "encryptionkey@0"
I20260808 15:56:15.931803 6006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:56:15.935051 6020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:15.936259 6006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 15:56:15.936424 6006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "2a2f6da1739949be8dd9f34612f60d8b"
format_stamp: "Formatted at 2026-08-08 15:56:15 on dist-test-slave-srgv"
server_key: "80ff54e1c6b1550c9d5f76abca9f4d94"
server_key_iv: "deff627c61a95ff2c5fc938c73ea1581"
server_key_version: "encryptionkey@0"
I20260808 15:56:15.936642 6006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:15.942212 6006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:15.942940 6006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:15.943246 6006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:15.963197 6006 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33745
I20260808 15:56:15.963229 6071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33745 every 8 connection(s)
I20260808 15:56:15.964493 6006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:56:15.968163 6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:56:15.969844 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6006
I20260808 15:56:15.970100 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:56:15.970913 28664 external_mini_cluster.cc:1428] Setting key aad57ecbec9b7f26b7755c81e0b567be
I20260808 15:56:15.979352 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:15.973031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.27.254.62:33745)
0808 15:56:15.974130 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:15.974140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:15.974182 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:56:15.975794 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:15.975815 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:15.976225 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:15.976552 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:15.977699 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:15.977704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:15.977717 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:15.977734 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:15.977793 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:15.977812 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:15.977990 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:15.978108 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 15:56:15.978425 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":421,"threads_started":1}
I20260808 15:56:15.981117 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b: Bootstrap starting.
I20260808 15:56:15.984396 6072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b: Neither blocks nor log segments found. Creating new log.
I20260808 15:56:15.985323 6072 log.cc:824] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:56:15.988209 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b: No bootstrap required, opened a new log
I20260808 15:56:15.995831 6072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2f6da1739949be8dd9f34612f60d8b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33745 } }
I20260808 15:56:15.996270 6072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:56:15.996376 6072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2f6da1739949be8dd9f34612f60d8b, State: Initialized, Role: FOLLOWER
I20260808 15:56:15.996989 6072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2f6da1739949be8dd9f34612f60d8b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33745 } }
I20260808 15:56:15.997223 6072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:56:15.997367 6072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:56:15.997581 6072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:56:15.999696 6072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2f6da1739949be8dd9f34612f60d8b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33745 } }
I20260808 15:56:16.000200 6072 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2f6da1739949be8dd9f34612f60d8b; no voters:
I20260808 15:56:16.000819 6072 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:56:16.000963 6077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:56:16.001716 6077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [term 1 LEADER]: Becoming Leader. State: Replica: 2a2f6da1739949be8dd9f34612f60d8b, State: Running, Role: LEADER
I20260808 15:56:16.002362 6077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2f6da1739949be8dd9f34612f60d8b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33745 } }
I20260808 15:56:16.002942 6072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:56:16.005084 6078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2f6da1739949be8dd9f34612f60d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2f6da1739949be8dd9f34612f60d8b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33745 } } }
I20260808 15:56:16.005807 6078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [sys.catalog]: This master's current role is: LEADER
I20260808 15:56:16.005472 6079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2f6da1739949be8dd9f34612f60d8b. Latest consensus state: current_term: 1 leader_uuid: "2a2f6da1739949be8dd9f34612f60d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2f6da1739949be8dd9f34612f60d8b" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33745 } } }
I20260808 15:56:16.006045 6079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b [sys.catalog]: This master's current role is: LEADER
I20260808 15:56:16.007741 6083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:56:16.012346 6083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:56:16.020740 6083 catalog_manager.cc:1347] Generated new cluster ID: fbf9ff29504742819d5f06f5803bd124
I20260808 15:56:16.020843 6083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:56:16.056299 6083 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:56:16.056712 6083 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:56:16.068990 6083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2f6da1739949be8dd9f34612f60d8b: Generated new TSK 0
I20260808 15:56:16.069864 6083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:56:16.079123 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33745
--builtin_ntp_servers=127.27.254.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 15:56:16.407863 6096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:16.408143 6096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:16.408355 6096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:16.415611 6096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:16.415707 6096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:16.415876 6096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:56:16.424865 6096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:16.426132 6096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:16.427873 6096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:16.441177 6104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:16.441434 6101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:16.441210 6102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:16.442291 6096 server_base.cc:1034] running on GCE node
I20260808 15:56:16.442910 6096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:16.443850 6096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:16.445075 6096 hybrid_clock.cc:648] HybridClock initialized: now 1786204576445030 us; error 54 us; skew 500 ppm
I20260808 15:56:16.445387 6096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:16.447577 6096 webserver.cc:460] Webserver started at http://127.27.254.1:41155/ using document root <none> and password file <none>
I20260808 15:56:16.448215 6096 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:16.448300 6096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:16.448570 6096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:16.451381 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "565b0cb5bd3e47a59003269f11ef9a1e"
format_stamp: "Formatted at 2026-08-08 15:56:16 on dist-test-slave-srgv"
server_key: "428abe584fdc05e5cc666ec0197968e3"
server_key_iv: "7b67f3caac2938ced613d52ab4268918"
server_key_version: "encryptionkey@0"
I20260808 15:56:16.452111 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "565b0cb5bd3e47a59003269f11ef9a1e"
format_stamp: "Formatted at 2026-08-08 15:56:16 on dist-test-slave-srgv"
server_key: "428abe584fdc05e5cc666ec0197968e3"
server_key_iv: "7b67f3caac2938ced613d52ab4268918"
server_key_version: "encryptionkey@0"
I20260808 15:56:16.457047 6096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:56:16.460363 6110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:16.461731 6096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:56:16.461890 6096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "565b0cb5bd3e47a59003269f11ef9a1e"
format_stamp: "Formatted at 2026-08-08 15:56:16 on dist-test-slave-srgv"
server_key: "428abe584fdc05e5cc666ec0197968e3"
server_key_iv: "7b67f3caac2938ced613d52ab4268918"
server_key_version: "encryptionkey@0"
I20260808 15:56:16.462103 6096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:16.467593 6096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:16.468338 6096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:16.468607 6096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:16.469527 6096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:16.470885 6096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:16.470992 6096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:16.471124 6096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:16.471226 6096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:16.503929 6096 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:46667
I20260808 15:56:16.504010 6222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:46667 every 8 connection(s)
I20260808 15:56:16.505275 6096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:56:16.514115 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:16.507950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:44733 (local address 127.27.254.62:33745)
0808 15:56:16.508389 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:16.508394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:16.509735 (+ 1341us) server_negotiation.cc:407] Connection header received
0808 15:56:16.511635 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:16.511641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:16.511719 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:16.511873 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:16.513285 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:16.513291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:16.513296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:16.513314 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:16.513366 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:16.513378 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:16.513554 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:16.513762 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 15:56:16.513833 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260808 15:56:16.514544 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6096
I20260808 15:56:16.514747 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:56:16.515125 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:16.508590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33745 (local address 127.27.254.1:44733)
0808 15:56:16.509582 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:16.509626 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:16.511362 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:16.512103 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:16.512111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:16.512204 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:16.512915 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:56:16.512919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:16.512980 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:56:16.513106 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:16.513517 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:16.513530 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:16.513539 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:56:16.513838 (+ 299us) client_negotiation.cc:240] Negotiation successful
0808 15:56:16.514100 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":302,"threads_started":1}
I20260808 15:56:16.515501 28664 external_mini_cluster.cc:1428] Setting key 68a0947265f62fcfe64c44ea335342c9
I20260808 15:56:16.516788 6223 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33745
I20260808 15:56:16.517099 6223 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:16.517875 6223 heartbeater.cc:511] Master 127.27.254.62:33745 requested a full tablet report, sending...
I20260808 15:56:16.518788 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33745
--builtin_ntp_servers=127.27.254.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 15:56:16.520157 6037 ts_manager.cc:194] Registered new tserver with Master: 565b0cb5bd3e47a59003269f11ef9a1e (127.27.254.1:46667)
W20260808 15:56:16.842336 6227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:16.842654 6227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:16.842808 6227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:16.850102 6227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:16.850240 6227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:16.850414 6227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:56:16.859651 6227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:16.860962 6227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:16.862821 6227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:16.875885 6232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:16.875960 6235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:16.875991 6233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:16.876888 6227 server_base.cc:1034] running on GCE node
I20260808 15:56:16.877554 6227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:16.878473 6227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:16.879706 6227 hybrid_clock.cc:648] HybridClock initialized: now 1786204576879652 us; error 58 us; skew 500 ppm
I20260808 15:56:16.880013 6227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:16.882324 6227 webserver.cc:460] Webserver started at http://127.27.254.2:43331/ using document root <none> and password file <none>
I20260808 15:56:16.882978 6227 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:16.883114 6227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:16.883447 6227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:16.886324 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "515256f85a424c2fa77a5545595dcd9b"
format_stamp: "Formatted at 2026-08-08 15:56:16 on dist-test-slave-srgv"
server_key: "aa049774d89b1f737f140e8d49c2cb64"
server_key_iv: "0caf486e85069af7731dc381fee2c5cf"
server_key_version: "encryptionkey@0"
I20260808 15:56:16.887111 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "515256f85a424c2fa77a5545595dcd9b"
format_stamp: "Formatted at 2026-08-08 15:56:16 on dist-test-slave-srgv"
server_key: "aa049774d89b1f737f140e8d49c2cb64"
server_key_iv: "0caf486e85069af7731dc381fee2c5cf"
server_key_version: "encryptionkey@0"
I20260808 15:56:16.892122 6227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:56:16.895501 6241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:16.896749 6227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:56:16.896920 6227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "515256f85a424c2fa77a5545595dcd9b"
format_stamp: "Formatted at 2026-08-08 15:56:16 on dist-test-slave-srgv"
server_key: "aa049774d89b1f737f140e8d49c2cb64"
server_key_iv: "0caf486e85069af7731dc381fee2c5cf"
server_key_version: "encryptionkey@0"
I20260808 15:56:16.897146 6227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:16.902742 6227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:16.903573 6227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:16.903831 6227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:16.904755 6227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:16.906142 6227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:16.906260 6227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:16.906385 6227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:16.906469 6227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:16.939450 6227 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:35355
I20260808 15:56:16.939535 6353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:35355 every 8 connection(s)
I20260808 15:56:16.940811 6227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:56:16.942289 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6227
I20260808 15:56:16.942502 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:56:16.942983 28664 external_mini_cluster.cc:1428] Setting key 802ebd5ef2b13559553e24a763e8e14e
I20260808 15:56:16.946115 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33745
--builtin_ntp_servers=127.27.254.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 15:56:16.950306 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:16.943657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:34351 (local address 127.27.254.62:33745)
0808 15:56:16.943862 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:16.943875 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:16.946222 (+ 2347us) server_negotiation.cc:407] Connection header received
0808 15:56:16.947979 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:16.947984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:16.948052 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:16.948175 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:16.949542 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:16.949549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:16.949560 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:16.949580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:16.949619 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:16.949630 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:16.949754 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:16.949968 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 15:56:16.950036 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:56:16.951213 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:16.944268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33745 (local address 127.27.254.2:34351)
0808 15:56:16.946029 (+ 1761us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:16.946084 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:16.947696 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:16.948425 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:16.948434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:16.948529 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:16.949222 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:56:16.949224 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:16.949261 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:56:16.949343 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:16.949727 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:16.949738 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:16.949745 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:56:16.950051 (+ 306us) client_negotiation.cc:240] Negotiation successful
0808 15:56:16.950313 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":1156,"threads_started":1}
I20260808 15:56:16.952561 6354 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33745
I20260808 15:56:16.952845 6354 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:16.953503 6354 heartbeater.cc:511] Master 127.27.254.62:33745 requested a full tablet report, sending...
I20260808 15:56:16.954974 6037 ts_manager.cc:194] Registered new tserver with Master: 515256f85a424c2fa77a5545595dcd9b (127.27.254.2:35355)
W20260808 15:56:17.272336 6357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:17.272589 6357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:17.272675 6357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:17.279852 6357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:17.279944 6357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:17.280156 6357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:56:17.288980 6357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:17.290338 6357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:17.292098 6357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:17.305208 6363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:17.305490 6362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:17.305430 6365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:17.306113 6357 server_base.cc:1034] running on GCE node
I20260808 15:56:17.306666 6357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:17.307641 6357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:17.308867 6357 hybrid_clock.cc:648] HybridClock initialized: now 1786204577308824 us; error 47 us; skew 500 ppm
I20260808 15:56:17.309180 6357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:56:17.311328 6357 webserver.cc:460] Webserver started at http://127.27.254.3:38685/ using document root <none> and password file <none>
I20260808 15:56:17.311995 6357 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:17.312160 6357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:17.312490 6357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:17.315307 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "8007974354084bb2944a5552386fd4b5"
format_stamp: "Formatted at 2026-08-08 15:56:17 on dist-test-slave-srgv"
server_key: "272980eae4d9683b4df9097048827723"
server_key_iv: "8518766d5d56c4d45f86a8aba941d244"
server_key_version: "encryptionkey@0"
I20260808 15:56:17.316093 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "8007974354084bb2944a5552386fd4b5"
format_stamp: "Formatted at 2026-08-08 15:56:17 on dist-test-slave-srgv"
server_key: "272980eae4d9683b4df9097048827723"
server_key_iv: "8518766d5d56c4d45f86a8aba941d244"
server_key_version: "encryptionkey@0"
I20260808 15:56:17.320907 6357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260808 15:56:17.324184 6371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:17.325488 6357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:56:17.325645 6357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "8007974354084bb2944a5552386fd4b5"
format_stamp: "Formatted at 2026-08-08 15:56:17 on dist-test-slave-srgv"
server_key: "272980eae4d9683b4df9097048827723"
server_key_iv: "8518766d5d56c4d45f86a8aba941d244"
server_key_version: "encryptionkey@0"
I20260808 15:56:17.325851 6357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:17.331349 6357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:17.332098 6357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:17.332360 6357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:17.333241 6357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:17.334604 6357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:17.334723 6357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:17.334854 6357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:17.334940 6357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:17.366710 6357 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:45365
I20260808 15:56:17.366751 6485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:45365 every 8 connection(s)
I20260808 15:56:17.368036 6357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:56:17.369585 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6357
I20260808 15:56:17.369779 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:56:17.370348 28664 external_mini_cluster.cc:1428] Setting key 0d03aac0cef3421167d3235a62a85d09
I20260808 15:56:17.376958 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:17.370998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:36409 (local address 127.27.254.62:33745)
0808 15:56:17.371217 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:17.371222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:17.372885 (+ 1663us) server_negotiation.cc:407] Connection header received
0808 15:56:17.374786 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:17.374791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:17.374856 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:17.374984 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:17.376221 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:17.376226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:17.376231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:17.376249 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:17.376293 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:17.376304 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:17.376423 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:17.376619 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 15:56:17.376684 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:56:17.377887 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:17.371649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33745 (local address 127.27.254.3:36409)
0808 15:56:17.372732 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:17.372766 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:17.374536 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:17.375197 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:17.375204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:17.375321 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:17.375941 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:56:17.375943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:17.375981 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:56:17.376060 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:17.376396 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:17.376407 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:17.376414 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:56:17.376695 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 15:56:17.376926 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":336,"threads_started":1}
I20260808 15:56:17.379052 6486 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33745
I20260808 15:56:17.379334 6486 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:17.379858 6486 heartbeater.cc:511] Master 127.27.254.62:33745 requested a full tablet report, sending...
I20260808 15:56:17.381168 6037 ts_manager.cc:194] Registered new tserver with Master: 8007974354084bb2944a5552386fd4b5 (127.27.254.3:45365)
I20260808 15:56:17.385660 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:56:17.392359 6225 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:56:17.390871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.27.254.62:33745)
0808 15:56:17.391034 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:17.391039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:17.391515 (+ 476us) server_negotiation.cc:407] Connection header received
0808 15:56:17.391651 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:17.391655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:17.391706 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:17.391783 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:17.392120 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58606: BlockingRecv error: recv got EOF from 127.0.0.1:58606 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:56:17.393010 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6096
I20260808 15:56:17.396482 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:17.394037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.27.254.1:46667)
0808 15:56:17.394585 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:17.394592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:17.394617 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:56:17.394671 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:17.394676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:17.394902 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:17.395022 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:17.395913 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:17.395919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:17.395930 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:17.395941 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:17.395980 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:17.395993 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:17.396110 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:17.396187 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 15:56:17.396264 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260808 15:56:17.397707 6218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:17.478502 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6096
I20260808 15:56:17.493801 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6227
I20260808 15:56:17.497380 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:17.494691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.27.254.2:35355)
0808 15:56:17.495187 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:17.495194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:17.495219 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:56:17.495279 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:17.495285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:17.495517 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:17.495681 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:17.496662 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:17.496668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:17.496679 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:17.496695 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:17.496733 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:17.496746 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:17.496885 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:17.497013 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:56:17.497117 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260808 15:56:17.498566 6349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:17.578507 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6227
I20260808 15:56:17.594609 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6357
I20260808 15:56:17.598456 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:17.595544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.27.254.3:45365)
0808 15:56:17.596211 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:17.596218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:17.596247 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:56:17.596304 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:17.596310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:17.596519 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:17.596652 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:17.597751 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:17.597757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:17.597768 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:56:17.597784 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:17.597821 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:56:17.597835 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:17.597963 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:17.598017 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 15:56:17.598125 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":331,"threads_started":1}
I20260808 15:56:17.599673 6481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:17.680051 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6357
I20260808 15:56:17.696972 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6006
I20260808 15:56:17.698470 6067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:17.799322 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6006
2026-08-08T15:56:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2310 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:56:17 dist-test-slave-srgv krb5kdc[6504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:56:17 dist-test-slave-srgv krb5kdc[6504](info): set up 2 sockets
Aug 08 15:56:17 dist-test-slave-srgv krb5kdc[6504](info): commencing operation
krb5kdc: starting...
W20260808 15:56:18.892665 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:56:19 dist-test-slave-srgv krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:56:19.126240 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40555
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:40555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:56:19.454623 6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:19.454982 6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:19.455169 6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:19.462728 6517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:56:19.462879 6517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:19.462985 6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:19.463093 6517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:56:19.463176 6517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:56:19.473088 6517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:40555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:19.474500 6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:19.475845 6517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:19.487485 6523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:19.487514 6525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:19.488142 6522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:19.488369 6517 server_base.cc:1034] running on GCE node
I20260808 15:56:19.489166 6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:19.490675 6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:19.491907 6517 hybrid_clock.cc:648] HybridClock initialized: now 1786204579491864 us; error 47 us; skew 500 ppm
I20260808 15:56:19.492256 6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:56:19 dist-test-slave-srgv krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204579, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:56:19.499101 6517 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:56:19.500687 6517 webserver.cc:460] Webserver started at http://127.27.254.62:45147/ using document root <none> and password file <none>
I20260808 15:56:19.501391 6517 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:19.501585 6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:19.501901 6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:19.504901 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "ab1434b96a914b5fac689fa356a598b5"
format_stamp: "Formatted at 2026-08-08 15:56:19 on dist-test-slave-srgv"
server_key: "672f520b424fbd660ff9079197dc5268"
server_key_iv: "300b9556fe890db8be9eabee4d85cf9f"
server_key_version: "encryptionkey@0"
I20260808 15:56:19.505744 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "ab1434b96a914b5fac689fa356a598b5"
format_stamp: "Formatted at 2026-08-08 15:56:19 on dist-test-slave-srgv"
server_key: "672f520b424fbd660ff9079197dc5268"
server_key_iv: "300b9556fe890db8be9eabee4d85cf9f"
server_key_version: "encryptionkey@0"
I20260808 15:56:19.510947 6517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:56:19.514293 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:19.515594 6517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:56:19.515762 6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "ab1434b96a914b5fac689fa356a598b5"
format_stamp: "Formatted at 2026-08-08 15:56:19 on dist-test-slave-srgv"
server_key: "672f520b424fbd660ff9079197dc5268"
server_key_iv: "300b9556fe890db8be9eabee4d85cf9f"
server_key_version: "encryptionkey@0"
I20260808 15:56:19.515951 6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:19.541096 6517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:19.545351 6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:19.545675 6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:19.564571 6517 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:40555
I20260808 15:56:19.564615 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:40555 every 8 connection(s)
I20260808 15:56:19.565944 6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:56:19.569772 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6517
I20260808 15:56:19.569859 6584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:56:19.570039 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:56:19.570887 28664 external_mini_cluster.cc:1428] Setting key 4d0578216865974c25d32dbbbdf67842
I20260808 15:56:19.583868 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5: Bootstrap starting.
Aug 08 15:56:19 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:56:19.587883 6584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5: Neither blocks nor log segments found. Creating new log.
I20260808 15:56:19.588979 6584 log.cc:824] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5: Log is configured to *not* fsync() on all Append() calls
I20260808 15:56:19.592276 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5: No bootstrap required, opened a new log
I20260808 15:56:19.595094 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:19.572986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.27.254.62:40555)
0808 15:56:19.573808 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:19.573820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:19.573860 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:56:19.576045 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:19.576067 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:19.576451 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:19.576785 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:19.578678 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:19.579997 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:19.581885 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:19.582525 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:19.582660 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:19.587633 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:19.587640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:19.587654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:19.587692 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:19.591085 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:19.591905 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:19.591912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:19.591917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:19.592096 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:19.592499 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:19.592502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:19.592504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:19.593009 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:19.593236 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:19.593708 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 15:56:19.594034 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":394,"threads_started":1}
I20260808 15:56:19.600505 6584 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1434b96a914b5fac689fa356a598b5" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40555 } }
I20260808 15:56:19.600908 6584 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:56:19.601053 6584 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1434b96a914b5fac689fa356a598b5, State: Initialized, Role: FOLLOWER
I20260808 15:56:19.601670 6584 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1434b96a914b5fac689fa356a598b5" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40555 } }
I20260808 15:56:19.601914 6584 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:56:19.602048 6584 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:56:19.602214 6584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:56:19.604391 6584 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1434b96a914b5fac689fa356a598b5" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40555 } }
I20260808 15:56:19.604914 6584 leader_election.cc:302] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab1434b96a914b5fac689fa356a598b5; no voters:
I20260808 15:56:19.605579 6584 leader_election.cc:288] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:56:19.605696 6589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:56:19.606434 6589 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [term 1 LEADER]: Becoming Leader. State: Replica: ab1434b96a914b5fac689fa356a598b5, State: Running, Role: LEADER
I20260808 15:56:19.606980 6589 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1434b96a914b5fac689fa356a598b5" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40555 } }
I20260808 15:56:19.607756 6584 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:56:19.609802 6591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1434b96a914b5fac689fa356a598b5. Latest consensus state: current_term: 1 leader_uuid: "ab1434b96a914b5fac689fa356a598b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1434b96a914b5fac689fa356a598b5" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40555 } } }
I20260808 15:56:19.610150 6591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [sys.catalog]: This master's current role is: LEADER
I20260808 15:56:19.610601 6590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab1434b96a914b5fac689fa356a598b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1434b96a914b5fac689fa356a598b5" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40555 } } }
I20260808 15:56:19.611038 6590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5 [sys.catalog]: This master's current role is: LEADER
I20260808 15:56:19.611227 6595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:56:19.623692 6595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:56:19.633296 6595 catalog_manager.cc:1347] Generated new cluster ID: 3e68991b1fdf4652b6e3823a5d09da55
I20260808 15:56:19.633448 6595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:56:19.649358 6595 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:56:19.650923 6595 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:56:19.661128 6595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5: Generated new TSK 0
I20260808 15:56:19.662138 6595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:56:19.733372 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40555
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:56:20.073835 6611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:20.074141 6611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:20.074318 6611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:20.081650 6611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:20.081760 6611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:20.081949 6611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:56:20.091979 6611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:20.093307 6611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:20.094660 6611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:20.107578 6616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:20.107477 6617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:20.107547 6619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:20.108438 6611 server_base.cc:1034] running on GCE node
I20260808 15:56:20.109053 6611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:20.110013 6611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:20.111228 6611 hybrid_clock.cc:648] HybridClock initialized: now 1786204580111172 us; error 47 us; skew 500 ppm
I20260808 15:56:20.111536 6611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:56:20 dist-test-slave-srgv krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:56:20.114547 6611 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:56:20.115862 6611 webserver.cc:460] Webserver started at http://127.27.254.1:38025/ using document root <none> and password file <none>
I20260808 15:56:20.116537 6611 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:20.116695 6611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:20.117054 6611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:20.120086 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "a7b17404d7274349a59fde9941b45395"
format_stamp: "Formatted at 2026-08-08 15:56:20 on dist-test-slave-srgv"
server_key: "2c76cf19ed7844740f4a53d68bcf74fa"
server_key_iv: "4d905d9f76c89556255ff55ad8c7d3ca"
server_key_version: "encryptionkey@0"
I20260808 15:56:20.120870 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "a7b17404d7274349a59fde9941b45395"
format_stamp: "Formatted at 2026-08-08 15:56:20 on dist-test-slave-srgv"
server_key: "2c76cf19ed7844740f4a53d68bcf74fa"
server_key_iv: "4d905d9f76c89556255ff55ad8c7d3ca"
server_key_version: "encryptionkey@0"
I20260808 15:56:20.125835 6611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:56:20.129127 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:20.130579 6611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:56:20.130753 6611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "a7b17404d7274349a59fde9941b45395"
format_stamp: "Formatted at 2026-08-08 15:56:20 on dist-test-slave-srgv"
server_key: "2c76cf19ed7844740f4a53d68bcf74fa"
server_key_iv: "4d905d9f76c89556255ff55ad8c7d3ca"
server_key_version: "encryptionkey@0"
I20260808 15:56:20.130950 6611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:20.154523 6611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:20.158373 6611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:20.158672 6611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:20.159610 6611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:20.161126 6611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:20.161221 6611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:20.161362 6611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:20.161445 6611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:20.193890 6611 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:41581
I20260808 15:56:20.193936 6742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:41581 every 8 connection(s)
I20260808 15:56:20.195142 6611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:56:20.200224 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6611
I20260808 15:56:20.200470 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:56:20.201095 28664 external_mini_cluster.cc:1428] Setting key 065ce533c7526e5e256079fca1e55ed0
Aug 08 15:56:20 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:56:20.215164 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:20.197653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:43327 (local address 127.27.254.62:40555)
0808 15:56:20.198102 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:20.198107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:20.198938 (+ 831us) server_negotiation.cc:407] Connection header received
0808 15:56:20.200785 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:20.200791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:20.200875 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:20.200991 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:20.202868 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.203996 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.205536 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.205942 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.206082 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:20.209623 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:20.209628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:20.209632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:20.209661 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:20.212203 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:20.212902 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:20.212907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:20.212909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:20.213024 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:20.213454 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:20.213461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:20.213466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:20.213733 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:20.213890 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:20.214588 (+ 698us) server_negotiation.cc:299] Negotiation successful
0808 15:56:20.214807 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":234,"threads_started":1}
I20260808 15:56:20.216102 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:20.198101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40555 (local address 127.27.254.1:43327)
0808 15:56:20.198777 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:20.198823 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:20.200499 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:20.201236 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:20.201244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:20.201915 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:20.202693 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:20.202716 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.204130 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.204135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:20.205342 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:20.205353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.206169 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.206176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:20.206409 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:20.207072 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:20.207094 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:20.209373 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:20.212339 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:56:20.212348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:56:20.212365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:56:20.212775 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:56:20.213128 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:56:20.213134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:56:20.213138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:56:20.213319 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:56:20.213865 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:20.213873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:20.214217 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 15:56:20.214470 (+ 253us) client_negotiation.cc:240] Negotiation successful
0808 15:56:20.214992 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":258,"threads_started":1}
I20260808 15:56:20.217592 6743 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40555
I20260808 15:56:20.217937 6743 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:20.218847 6743 heartbeater.cc:511] Master 127.27.254.62:40555 requested a full tablet report, sending...
I20260808 15:56:20.221120 6549 ts_manager.cc:194] Registered new tserver with Master: a7b17404d7274349a59fde9941b45395 (127.27.254.1:41581)
I20260808 15:56:20.222883 6549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:43327
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:56:20.250555 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40555
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:56:20.577571 6749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:20.577875 6749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:20.578034 6749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:20.585387 6749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:20.585614 6749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:20.585791 6749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:56:20.595808 6749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:20.597162 6749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:20.598532 6749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:20.611711 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:20.611928 6757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:20.611979 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:20.612931 6749 server_base.cc:1034] running on GCE node
I20260808 15:56:20.613598 6749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:20.614516 6749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:20.615751 6749 hybrid_clock.cc:648] HybridClock initialized: now 1786204580615712 us; error 52 us; skew 500 ppm
I20260808 15:56:20.616080 6749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:56:20 dist-test-slave-srgv krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:56:20.619230 6749 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:56:20.620760 6749 webserver.cc:460] Webserver started at http://127.27.254.2:33629/ using document root <none> and password file <none>
I20260808 15:56:20.621518 6749 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:20.621672 6749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:20.621968 6749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:20.624847 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "9bc76879540b411fbab3c01fcf06d069"
format_stamp: "Formatted at 2026-08-08 15:56:20 on dist-test-slave-srgv"
server_key: "c3fed47be9b05d1ea85537ad3bc95878"
server_key_iv: "9eca1e5ae86984404438ecc8e13f8b45"
server_key_version: "encryptionkey@0"
I20260808 15:56:20.625674 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "9bc76879540b411fbab3c01fcf06d069"
format_stamp: "Formatted at 2026-08-08 15:56:20 on dist-test-slave-srgv"
server_key: "c3fed47be9b05d1ea85537ad3bc95878"
server_key_iv: "9eca1e5ae86984404438ecc8e13f8b45"
server_key_version: "encryptionkey@0"
I20260808 15:56:20.630828 6749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:56:20.634421 6764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:20.635807 6749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:56:20.635983 6749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "9bc76879540b411fbab3c01fcf06d069"
format_stamp: "Formatted at 2026-08-08 15:56:20 on dist-test-slave-srgv"
server_key: "c3fed47be9b05d1ea85537ad3bc95878"
server_key_iv: "9eca1e5ae86984404438ecc8e13f8b45"
server_key_version: "encryptionkey@0"
I20260808 15:56:20.636186 6749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:20.668272 6749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:20.672421 6749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:20.672729 6749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:20.673800 6749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:20.675161 6749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:20.675266 6749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:20.675417 6749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:20.675489 6749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:20.709091 6749 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:33491
I20260808 15:56:20.709138 6876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:33491 every 8 connection(s)
I20260808 15:56:20.710448 6749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:56:20.713001 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6749
I20260808 15:56:20.713233 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:56:20.713778 28664 external_mini_cluster.cc:1428] Setting key e9d4fe51c39a7734827f1d8711e37252
Aug 08 15:56:20 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204580, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:56:20.732234 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:20.713034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:57407 (local address 127.27.254.62:40555)
0808 15:56:20.713572 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:20.713577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:20.714772 (+ 1195us) server_negotiation.cc:407] Connection header received
0808 15:56:20.717047 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:20.717053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:20.717135 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:20.717248 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:20.718778 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.719626 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.721228 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.721720 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.721847 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:20.725215 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:20.725224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:20.725232 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:20.725277 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:20.728905 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:20.729644 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:20.729652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:20.729659 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:20.729815 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:20.730263 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:20.730270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:20.730276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:20.730659 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:20.730829 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:20.731482 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 15:56:20.731761 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260808 15:56:20.732822 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:20.713771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40555 (local address 127.27.254.2:57407)
0808 15:56:20.714618 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:20.714667 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:20.716786 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:20.717540 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:20.717549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:20.718049 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:20.718614 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:20.718632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.719772 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.719778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:20.721090 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:20.721102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:20.721950 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:20.721957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:20.722178 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:20.722876 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:20.722902 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:20.724974 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:20.729043 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:56:20.729049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:56:20.729060 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:56:20.729531 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:56:20.729925 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:56:20.729928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:56:20.729930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:56:20.730144 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:56:20.730786 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:20.730792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:20.731107 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 15:56:20.731330 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 15:56:20.731705 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":305,"threads_started":1}
I20260808 15:56:20.734355 6877 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40555
I20260808 15:56:20.734721 6877 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:20.735711 6877 heartbeater.cc:511] Master 127.27.254.62:40555 requested a full tablet report, sending...
I20260808 15:56:20.737375 6549 ts_manager.cc:194] Registered new tserver with Master: 9bc76879540b411fbab3c01fcf06d069 (127.27.254.2:33491)
I20260808 15:56:20.738260 6549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:57407
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:56:20.767819 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40555
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:56:21.092787 6882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:56:21.093055 6882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:56:21.093145 6882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:56:21.101161 6882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:56:21.101260 6882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:56:21.101366 6882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:56:21.110777 6882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:56:21.111955 6882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:56:21.113559 6882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:56:21.126142 6887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:21.126113 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:56:21.126070 6888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:56:21.127401 6882 server_base.cc:1034] running on GCE node
I20260808 15:56:21.127964 6882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:56:21.128898 6882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:56:21.130115 6882 hybrid_clock.cc:648] HybridClock initialized: now 1786204581130067 us; error 44 us; skew 500 ppm
I20260808 15:56:21.130522 6882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:56:21 dist-test-slave-srgv krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204581, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:56:21.133561 6882 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:56:21.134994 6882 webserver.cc:460] Webserver started at http://127.27.254.3:44639/ using document root <none> and password file <none>
I20260808 15:56:21.135668 6882 fs_manager.cc:359] Metadata directory not provided
I20260808 15:56:21.135764 6882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:56:21.135998 6882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:56:21.138737 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "26abda7669954c8ebfe10e9ff98d2076"
format_stamp: "Formatted at 2026-08-08 15:56:21 on dist-test-slave-srgv"
server_key: "7c4a2021fe8ca776c2ce909c69ed2e0e"
server_key_iv: "99b87128b1dc7cf188a11c8ca3a2f5c8"
server_key_version: "encryptionkey@0"
I20260808 15:56:21.139518 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "26abda7669954c8ebfe10e9ff98d2076"
format_stamp: "Formatted at 2026-08-08 15:56:21 on dist-test-slave-srgv"
server_key: "7c4a2021fe8ca776c2ce909c69ed2e0e"
server_key_iv: "99b87128b1dc7cf188a11c8ca3a2f5c8"
server_key_version: "encryptionkey@0"
I20260808 15:56:21.144603 6882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:56:21.147981 6897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:21.149267 6882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:56:21.149397 6882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "26abda7669954c8ebfe10e9ff98d2076"
format_stamp: "Formatted at 2026-08-08 15:56:21 on dist-test-slave-srgv"
server_key: "7c4a2021fe8ca776c2ce909c69ed2e0e"
server_key_iv: "99b87128b1dc7cf188a11c8ca3a2f5c8"
server_key_version: "encryptionkey@0"
I20260808 15:56:21.149610 6882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:56:21.169309 6882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:56:21.173009 6882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:56:21.173251 6882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:56:21.174151 6882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:56:21.175414 6882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:56:21.175496 6882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:21.175583 6882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:56:21.175626 6882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:56:21.208014 6882 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:36073
I20260808 15:56:21.208098 7010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:36073 every 8 connection(s)
I20260808 15:56:21.209224 6882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:56:21.218644 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 6882
I20260808 15:56:21.218837 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:56:21.219396 28664 external_mini_cluster.cc:1428] Setting key 56600a0bd4a68d5ce8e4bab643c70424
Aug 08 15:56:21 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204581, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:56:21.226015 6743 heartbeater.cc:503] Master 127.27.254.62:40555 was elected leader, sending a full tablet report...
I20260808 15:56:21.229074 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.211737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:54365 (local address 127.27.254.62:40555)
0808 15:56:21.211938 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.211944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.213145 (+ 1201us) server_negotiation.cc:407] Connection header received
0808 15:56:21.214919 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.214923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.214988 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.215102 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:21.216472 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.217225 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.218693 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.219527 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.219661 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.223475 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:21.223481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:21.223486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:21.223546 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:21.226266 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.227029 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.227032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.227035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.227165 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.227658 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.227661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.227663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.227918 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:21.228043 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.228675 (+ 632us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.228836 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:56:21.230110 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.212320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40555 (local address 127.27.254.3:54365)
0808 15:56:21.212988 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:21.213026 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:21.214659 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:21.215301 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:21.215309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:21.215806 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:56:21.216332 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.216348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.217347 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.217350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.218558 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.218570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.219727 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.219731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.220004 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.220736 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:56:21.220785 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:56:21.223277 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:56:21.226392 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:56:21.226398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:56:21.226410 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:56:21.226907 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:56:21.227305 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:56:21.227309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:56:21.227312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:56:21.227529 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:56:21.228028 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:56:21.228033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:56:21.228334 (+ 301us) client_negotiation.cc:769] Sending connection context
0808 15:56:21.228557 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 15:56:21.228929 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260808 15:56:21.231395 7011 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40555
I20260808 15:56:21.231685 7011 heartbeater.cc:464] Registering TS with master...
I20260808 15:56:21.232375 7011 heartbeater.cc:511] Master 127.27.254.62:40555 requested a full tablet report, sending...
I20260808 15:56:21.233856 6549 ts_manager.cc:194] Registered new tserver with Master: 26abda7669954c8ebfe10e9ff98d2076 (127.27.254.3:36073)
I20260808 15:56:21.234881 6549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:54365
I20260808 15:56:21.235045 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:56:21.251020 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.239400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.27.254.62:40555)
0808 15:56:21.239536 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.239540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.240020 (+ 480us) server_negotiation.cc:407] Connection header received
0808 15:56:21.240148 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.240151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.240203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.240278 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:21.241733 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.242425 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.243629 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.244022 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.244114 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.246287 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:21.246291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:21.246295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:21.246321 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:21.248580 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.249184 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.249187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.249189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.249309 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.249724 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.249727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.249729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.250037 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:21.250214 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.250491 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.250667 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 15:56:21.254808 6549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:56:21.261324 6549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:56:21.294325 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.272776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.27.254.2:33491)
0808 15:56:21.284800 (+ 12024us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.284813 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.284853 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:56:21.286050 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.286070 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.286360 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.286572 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:56:21.288030 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.289315 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.291719 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.293037 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.293189 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.293393 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.293809 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.293984 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11800,"thread_start_us":11684,"threads_started":1}
I20260808 15:56:21.294400 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.272373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:33491 (local address 127.0.0.1:37028)
0808 15:56:21.273210 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:21.273252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:21.273397 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:21.286866 (+ 13469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:21.286877 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:21.286913 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:56:21.287825 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.287842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.289426 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.289431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.291556 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.291569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.293268 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.293276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.293540 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.293600 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 15:56:21.293736 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 15:56:21.293936 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":416,"spinlock_wait_cycles":6912,"thread_start_us":582,"threads_started":1}
I20260808 15:56:21.295130 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.272632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41581 (local address 127.0.0.1:38338)
0808 15:56:21.286083 (+ 13451us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:21.286104 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:21.286238 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:21.287180 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:21.287186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:21.287216 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:56:21.287772 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.287807 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.289820 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.289826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.292033 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.292046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.294228 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.294235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.294423 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.294470 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 15:56:21.294571 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:56:21.294687 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13328,"mutex_wait_us":12,"thread_start_us":13195,"threads_started":1}
I20260808 15:56:21.295991 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.273265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.27.254.1:41581)
0808 15:56:21.285423 (+ 12158us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.285436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.286365 (+ 929us) server_negotiation.cc:407] Connection header received
0808 15:56:21.286473 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.286483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.286740 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.286924 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:56:21.288026 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.289670 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.292208 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.293671 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.293827 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.294075 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.294894 (+ 819us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.295643 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11956,"thread_start_us":12159,"threads_started":1}
I20260808 15:56:21.296713 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.273734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:36073 (local address 127.0.0.1:41790)
0808 15:56:21.287375 (+ 13641us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:21.287408 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:21.287562 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:21.288767 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:21.288772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:21.288798 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:56:21.289164 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.289175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.291190 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.291198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.293370 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.293387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.295939 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.295947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.296151 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.296189 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:56:21.296295 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:56:21.296425 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13422,"thread_start_us":13929,"threads_started":1}
I20260808 15:56:21.296856 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.274110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.27.254.3:36073)
0808 15:56:21.287924 (+ 13814us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.287932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.287964 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:56:21.288066 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.288073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.288304 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.288487 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:56:21.289381 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.291004 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.293575 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.295045 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.295165 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.295401 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.296346 (+ 945us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.296523 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13630,"thread_start_us":13459,"threads_started":1}
I20260808 15:56:21.299032 6812 tablet_service.cc:1467] Processing CreateTablet for tablet 1edc0e70667b4e3398e21d71c4605d44 (DEFAULT_TABLE table=test-table [id=7a3c727d7817472ea56d1f144e99b040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:56:21.299656 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 1edc0e70667b4e3398e21d71c4605d44 (DEFAULT_TABLE table=test-table [id=7a3c727d7817472ea56d1f144e99b040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:56:21.300554 6812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edc0e70667b4e3398e21d71c4605d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:56:21.300220 6946 tablet_service.cc:1467] Processing CreateTablet for tablet 1edc0e70667b4e3398e21d71c4605d44 (DEFAULT_TABLE table=test-table [id=7a3c727d7817472ea56d1f144e99b040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:56:21.301064 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edc0e70667b4e3398e21d71c4605d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:56:21.301509 6946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edc0e70667b4e3398e21d71c4605d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:56:21.312520 7030 tablet_bootstrap.cc:492] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Bootstrap starting.
I20260808 15:56:21.315166 7031 tablet_bootstrap.cc:492] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Bootstrap starting.
I20260808 15:56:21.315948 7030 tablet_bootstrap.cc:654] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Neither blocks nor log segments found. Creating new log.
I20260808 15:56:21.316829 7032 tablet_bootstrap.cc:492] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Bootstrap starting.
I20260808 15:56:21.317665 7030 log.cc:824] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Log is configured to *not* fsync() on all Append() calls
I20260808 15:56:21.319027 7031 tablet_bootstrap.cc:654] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Neither blocks nor log segments found. Creating new log.
I20260808 15:56:21.320109 7032 tablet_bootstrap.cc:654] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Neither blocks nor log segments found. Creating new log.
I20260808 15:56:21.320570 7031 log.cc:824] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Log is configured to *not* fsync() on all Append() calls
I20260808 15:56:21.321516 7032 log.cc:824] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Log is configured to *not* fsync() on all Append() calls
I20260808 15:56:21.323772 7031 tablet_bootstrap.cc:492] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: No bootstrap required, opened a new log
I20260808 15:56:21.324047 7031 ts_tablet_manager.cc:1397] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260808 15:56:21.324528 7032 tablet_bootstrap.cc:492] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: No bootstrap required, opened a new log
I20260808 15:56:21.324653 7030 tablet_bootstrap.cc:492] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: No bootstrap required, opened a new log
I20260808 15:56:21.324790 7032 ts_tablet_manager.cc:1397] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 15:56:21.324894 7030 ts_tablet_manager.cc:1397] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260808 15:56:21.331498 7031 raft_consensus.cc:348] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.331856 7031 raft_consensus.cc:374] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:56:21.331977 7031 raft_consensus.cc:729] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b17404d7274349a59fde9941b45395, State: Initialized, Role: FOLLOWER
I20260808 15:56:21.332543 7031 consensus_queue.cc:260] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.334283 7031 ts_tablet_manager.cc:1428] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260808 15:56:21.335510 7032 raft_consensus.cc:348] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.335497 7030 raft_consensus.cc:348] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.335922 7030 raft_consensus.cc:374] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:56:21.335924 7032 raft_consensus.cc:374] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:56:21.336031 7030 raft_consensus.cc:729] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26abda7669954c8ebfe10e9ff98d2076, State: Initialized, Role: FOLLOWER
I20260808 15:56:21.336032 7032 raft_consensus.cc:729] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc76879540b411fbab3c01fcf06d069, State: Initialized, Role: FOLLOWER
I20260808 15:56:21.336642 7030 consensus_queue.cc:260] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.336653 7032 consensus_queue.cc:260] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.338132 6877 heartbeater.cc:503] Master 127.27.254.62:40555 was elected leader, sending a full tablet report...
I20260808 15:56:21.338583 7011 heartbeater.cc:503] Master 127.27.254.62:40555 was elected leader, sending a full tablet report...
I20260808 15:56:21.338929 7032 ts_tablet_manager.cc:1428] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260808 15:56:21.339159 7030 ts_tablet_manager.cc:1428] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Time spent starting tablet: real 0.014s user 0.003s sys 0.012s
W20260808 15:56:21.447510 6744 tablet.cc:2369] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:56:21.460975 7012 tablet.cc:2369] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:56:21.462486 6878 tablet.cc:2369] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:56:21.587137 7037 raft_consensus.cc:482] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:56:21.587401 7037 raft_consensus.cc:504] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.588980 7037 leader_election.cc:288] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7b17404d7274349a59fde9941b45395 (127.27.254.1:41581), 26abda7669954c8ebfe10e9ff98d2076 (127.27.254.3:36073)
I20260808 15:56:21.598245 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.589601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:59351 (local address 127.27.254.1:41581)
0808 15:56:21.589832 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.589841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.590369 (+ 528us) server_negotiation.cc:407] Connection header received
0808 15:56:21.590608 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.590613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.590692 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.590793 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:56:21.591654 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.592725 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.595821 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.597042 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.597220 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.597290 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.597841 (+ 551us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.597980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:56:21.598233 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.589582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:48055 (local address 127.27.254.3:36073)
0808 15:56:21.589809 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.589816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.590474 (+ 658us) server_negotiation.cc:407] Connection header received
0808 15:56:21.590610 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.590614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.590692 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.590793 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:56:21.591560 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.592725 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.595821 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.597042 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.597215 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.597290 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.597841 (+ 551us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.598006 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:56:21.598618 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.589397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:36073 (local address 127.27.254.2:48055)
0808 15:56:21.590374 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:21.590389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:21.590488 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:21.590822 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:21.590827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:21.590871 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:56:21.591368 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.591387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.592858 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.592863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.595640 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.595657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.597337 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.597342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.597597 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.597662 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 15:56:21.597764 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:56:21.597951 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":270,"spinlock_wait_cycles":8320,"thread_start_us":606,"threads_started":1}
I20260808 15:56:21.598644 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.589346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41581 (local address 127.27.254.2:59351)
0808 15:56:21.590249 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0808 15:56:21.590270 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:56:21.590439 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:56:21.590822 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:56:21.590827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:56:21.590864 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:56:21.591490 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.591517 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.592861 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.592865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.595640 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:56:21.595657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.597335 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.597342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:56:21.597598 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.597672 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 15:56:21.597764 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:56:21.597951 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":312,"spinlock_wait_cycles":1408,"thread_start_us":508,"threads_started":1}
I20260808 15:56:21.599936 6697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0e70667b4e3398e21d71c4605d44" candidate_uuid: "9bc76879540b411fbab3c01fcf06d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7b17404d7274349a59fde9941b45395" is_pre_election: true
I20260808 15:56:21.600020 6966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0e70667b4e3398e21d71c4605d44" candidate_uuid: "9bc76879540b411fbab3c01fcf06d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abda7669954c8ebfe10e9ff98d2076" is_pre_election: true
I20260808 15:56:21.600485 6966 raft_consensus.cc:2454] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc76879540b411fbab3c01fcf06d069 in term 0.
I20260808 15:56:21.600482 6697 raft_consensus.cc:2454] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc76879540b411fbab3c01fcf06d069 in term 0.
I20260808 15:56:21.601136 6768 leader_election.cc:302] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26abda7669954c8ebfe10e9ff98d2076, 9bc76879540b411fbab3c01fcf06d069, a7b17404d7274349a59fde9941b45395; no voters:
I20260808 15:56:21.601586 7037 raft_consensus.cc:2781] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:56:21.601727 7037 raft_consensus.cc:482] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:56:21.601843 7037 raft_consensus.cc:3037] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:56:21.603971 7037 raft_consensus.cc:504] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.604636 7037 leader_election.cc:288] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [CANDIDATE]: Term 1 election: Requested vote from peers a7b17404d7274349a59fde9941b45395 (127.27.254.1:41581), 26abda7669954c8ebfe10e9ff98d2076 (127.27.254.3:36073)
I20260808 15:56:21.605039 6697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0e70667b4e3398e21d71c4605d44" candidate_uuid: "9bc76879540b411fbab3c01fcf06d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7b17404d7274349a59fde9941b45395"
I20260808 15:56:21.605129 6966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0e70667b4e3398e21d71c4605d44" candidate_uuid: "9bc76879540b411fbab3c01fcf06d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26abda7669954c8ebfe10e9ff98d2076"
I20260808 15:56:21.605259 6697 raft_consensus.cc:3037] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:56:21.605309 6966 raft_consensus.cc:3037] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:56:21.608498 6697 raft_consensus.cc:2454] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc76879540b411fbab3c01fcf06d069 in term 1.
I20260808 15:56:21.608510 6966 raft_consensus.cc:2454] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc76879540b411fbab3c01fcf06d069 in term 1.
I20260808 15:56:21.608932 6768 leader_election.cc:302] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26abda7669954c8ebfe10e9ff98d2076, 9bc76879540b411fbab3c01fcf06d069, a7b17404d7274349a59fde9941b45395; no voters:
I20260808 15:56:21.609230 7037 raft_consensus.cc:2781] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:56:21.609717 7037 raft_consensus.cc:686] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc76879540b411fbab3c01fcf06d069, State: Running, Role: LEADER
I20260808 15:56:21.610184 7037 consensus_queue.cc:237] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } }
I20260808 15:56:21.614429 6549 catalog_manager.cc:5495] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc76879540b411fbab3c01fcf06d069 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "9bc76879540b411fbab3c01fcf06d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc76879540b411fbab3c01fcf06d069" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260808 15:56:21.662681 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.657815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.27.254.2:33491)
0808 15:56:21.657993 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.657999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.658114 (+ 115us) server_negotiation.cc:407] Connection header received
0808 15:56:21.658311 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.658315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.658392 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.658508 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:56:21.658808 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.659533 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.660756 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.661149 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.661236 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.661686 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:56:21.662222 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:56:21.662328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.662393 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.662455 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 15:56:21.670418 6966 raft_consensus.cc:1261] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 1 FOLLOWER]: Refusing update from remote peer 9bc76879540b411fbab3c01fcf06d069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:56:21.670496 6697 raft_consensus.cc:1261] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 1 FOLLOWER]: Refusing update from remote peer 9bc76879540b411fbab3c01fcf06d069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:56:21.671402 7041 consensus_queue.cc:1035] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7b17404d7274349a59fde9941b45395" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:56:21.671763 7037 consensus_queue.cc:1035] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26abda7669954c8ebfe10e9ff98d2076" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:56:21.685204 7046 mvcc.cc:205] Tried to move back new op lower bound from 7316293966513053696 to 7316293966278901760. Current Snapshot: MvccSnapshot[applied={T|T < 7316293966513053696 or (T in {7316293966513053696})}]
I20260808 15:56:21.699743 6549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:56:21.699990 6549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:56:21.703402 6549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab1434b96a914b5fac689fa356a598b5: Sending DeleteTablet for 3 replicas of tablet 1edc0e70667b4e3398e21d71c4605d44
I20260808 15:56:21.704506 6946 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edc0e70667b4e3398e21d71c4605d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:56:21 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:41790
I20260808 15:56:21.704506 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edc0e70667b4e3398e21d71c4605d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:56:21 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:38338
I20260808 15:56:21.704547 6812 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edc0e70667b4e3398e21d71c4605d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:56:21 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37028
I20260808 15:56:21.717789 7054 tablet_replica.cc:330] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: stopping tablet replica
I20260808 15:56:21.718272 7054 raft_consensus.cc:2229] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:56:21.718813 7054 raft_consensus.cc:2258] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:56:21.719142 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6611
I20260808 15:56:21.719733 7055 tablet_replica.cc:330] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: stopping tablet replica
I20260808 15:56:21.720178 7055 raft_consensus.cc:2229] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:56:21.720757 7055 raft_consensus.cc:2258] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:56:21.723125 7055 ts_tablet_manager.cc:1905] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:56:21.731927 7054 ts_tablet_manager.cc:1905] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:56:21.738278 7055 ts_tablet_manager.cc:1918] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:56:21.738435 7055 log.cc:1196] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/1edc0e70667b4e3398e21d71c4605d44
I20260808 15:56:21.738960 7055 ts_tablet_manager.cc:1939] T 1edc0e70667b4e3398e21d71c4605d44 P a7b17404d7274349a59fde9941b45395: Deleting consensus metadata
I20260808 15:56:21.740828 6533 catalog_manager.cc:4841] TS a7b17404d7274349a59fde9941b45395 (127.27.254.1:41581): tablet 1edc0e70667b4e3398e21d71c4605d44 (table test-table [id=7a3c727d7817472ea56d1f144e99b040]) successfully deleted
Aug 08 15:56:21 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:56:21.745817 7054 ts_tablet_manager.cc:1918] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:56:21.745995 7054 log.cc:1196] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/1edc0e70667b4e3398e21d71c4605d44
I20260808 15:56:21.746484 7054 ts_tablet_manager.cc:1939] T 1edc0e70667b4e3398e21d71c4605d44 P 26abda7669954c8ebfe10e9ff98d2076: Deleting consensus metadata
I20260808 15:56:21.748363 6536 catalog_manager.cc:4841] TS 26abda7669954c8ebfe10e9ff98d2076 (127.27.254.3:36073): tablet 1edc0e70667b4e3398e21d71c4605d44 (table test-table [id=7a3c727d7817472ea56d1f144e99b040]) successfully deleted
I20260808 15:56:21.748826 7056 tablet_replica.cc:330] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: stopping tablet replica
I20260808 15:56:21.749238 7056 raft_consensus.cc:2229] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:56:21.749907 7056 raft_consensus.cc:2258] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:56:21.752326 7056 ts_tablet_manager.cc:1905] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:56:21.752597 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.722200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.27.254.1:41581)
0808 15:56:21.722363 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.722369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.730112 (+ 7743us) server_negotiation.cc:407] Connection header received
0808 15:56:21.730179 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.730185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.730268 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.730372 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:21.732211 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.733314 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.735190 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.735799 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.735899 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.744709 (+ 8810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:21.744723 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:21.744739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:21.744777 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:21.748522 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.749634 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.749644 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.749647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.749815 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.750313 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.750318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.750322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.750799 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:21.751104 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.751815 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.752088 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:56:21.754231 6738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:21.759682 7056 ts_tablet_manager.cc:1918] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:56:21.759850 7056 log.cc:1196] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/1edc0e70667b4e3398e21d71c4605d44
I20260808 15:56:21.760380 7056 ts_tablet_manager.cc:1939] T 1edc0e70667b4e3398e21d71c4605d44 P 9bc76879540b411fbab3c01fcf06d069: Deleting consensus metadata
I20260808 15:56:21.762094 6536 catalog_manager.cc:4841] TS 9bc76879540b411fbab3c01fcf06d069 (127.27.254.2:33491): tablet 1edc0e70667b4e3398e21d71c4605d44 (table test-table [id=7a3c727d7817472ea56d1f144e99b040]) successfully deleted
I20260808 15:56:21.892987 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6611
I20260808 15:56:21.910887 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6749
Aug 08 15:56:21 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:56:21.927805 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:21.911666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.27.254.2:33491)
0808 15:56:21.911840 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:21.911846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:21.911893 (+ 47us) server_negotiation.cc:407] Connection header received
0808 15:56:21.912070 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:21.912076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:21.912181 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:21.912290 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:21.913450 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.914683 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.915885 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:21.916573 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:21.916706 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:21.920946 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:21.920963 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:21.920984 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:21.921031 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:21.925021 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.925710 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.925725 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.925729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.925895 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:21.926223 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:21.926229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:21.926234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:21.926665 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:21.927003 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:21.927176 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 15:56:21.927396 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:56:21.928510 6872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:22.047154 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6749
I20260808 15:56:22.073841 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6882
Aug 08 15:56:22 dist-test-slave-srgv krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:56:22.092567 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:56:22.074622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.27.254.3:36073)
0808 15:56:22.074841 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:56:22.074850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:56:22.074873 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:56:22.075050 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:56:22.075056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:56:22.075151 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:56:22.075277 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:56:22.076571 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:22.077275 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:22.078621 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:56:22.079370 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:56:22.079507 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:56:22.085131 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:56:22.085154 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:56:22.085182 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:56:22.085217 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:56:22.089095 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:22.089986 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:22.089998 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:22.090004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:22.090205 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:56:22.090618 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:56:22.090625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:56:22.090631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:56:22.091175 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:56:22.091553 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0808 15:56:22.091823 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 15:56:22.092094 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:56:22.093320 7006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:22.177734 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6882
I20260808 15:56:22.194763 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 6517
I20260808 15:56:22.195899 6579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:56:22.301849 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 6517
2026-08-08T15:56:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4503 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:56:22 dist-test-slave-srgv krb5kdc[7064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:56:22 dist-test-slave-srgv krb5kdc[7064](info): set up 2 sockets
Aug 08 15:56:22 dist-test-slave-srgv krb5kdc[7064](info): commencing operation
krb5kdc: starting...
W20260808 15:56:23.372145 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:56:23 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:56:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/rangeradmin_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/rangeradmin_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/rangerlookup_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/rangerlookup_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
I20260808 15:56:23.691507 28664 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBr1Jv8/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/postgres -l logfile start
2026-08-08 15:56:26.007 UTC [7094] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 15:56:26.007 UTC [7094] LOG: listening on IPv4 address "127.27.254.20", port 59029
2026-08-08 15:56:26.010 UTC [7094] LOG: listening on Unix socket "/tmp/.s.PGSQL.59029"
2026-08-08 15:56:26.032 UTC [7097] LOG: database system was shut down at 2026-08-08 15:56:24 UTC
2026-08-08 15:56:26.035 UTC [7094] LOG: database system is ready to accept connections
I20260808 15:56:26.991772 28664 mini_postgres.cc:96] Postgres bound to 59029
2026-08-08 15:56:27.010 UTC [7105] FATAL: database "slave" does not exist
127.27.254.20:59029 - accepting connections
I20260808 15:56:27.010902 28664 mini_ranger.cc:161] Starting Ranger...
I20260808 15:56:27.038249 28664 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 15:56:27.477916 28664 mini_ranger.cc:87] Created ranger Postgres database
I20260808 15:56:27.478056 28664 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-admin
2026-08-08 15:56:27,767 [I] DB FLAVOR :POSTGRES
2026-08-08 15:56:27,768 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:56:27,768 [I] Checking connection..
2026-08-08 15:56:27,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 15:56:28,022 [I] Checking connection passed.
2026-08-08 15:56:28,023 [I] --------- Verifying version history table ---------
2026-08-08 15:56:28,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:56:28,260 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:56:28,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:56:28,509 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:56:28,510 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 15:56:28,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 15:56:28.740 UTC [7169] WARNING: there is no transaction in progress
2026-08-08 15:56:28,746 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 15:56:28,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:56:28,995 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 15:56:28,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:56:28,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 15:56:29,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 15:56:29,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srgv.c.gcp-upstream.internal','N') ;"
2026-08-08 15:56:29,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:56:29,963 [I] Table x_portal_user does not exist in database ranger
2026-08-08 15:56:29,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:56:30,219 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 15:56:30,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:56:30,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:56:30,719 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 15:56:30,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 15:56:31.611 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.625 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.641 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.655 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.670 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.690 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.752 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.759 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.772 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.783 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.793 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.806 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.815 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.821 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.828 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:31.834 UTC [7278] WARNING: there is no transaction in progress
2026-08-08 15:56:32,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 15:56:32,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 15:56:32,646 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 15:56:32,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:56:32,886 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 15:56:32,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:56:33,142 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 15:56:33,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:56:33,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:56:33,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srgv.c.gcp-upstream.internal';"
2026-08-08 15:56:33,875 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 15:56:33,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 15:56:34,115 [I] DB_PATCHES have already been applied
I20260808 15:56:34.121094 28664 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBr1Jv8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 15:56:34.121172 28664 mini_ranger.cc:193] Using host: 127.27.254.20
I20260808 15:56:34.137069 28664 mini_ranger.cc:227] Ranger admin URL: http://127.27.254.20:41765
Aug 08, 2026 3:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 15:56:35.257789 28664 mini_ranger.cc:160] Time spent starting Ranger: real 8.247s user 0.003s sys 0.039s
Aug 08, 2026 3:56:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:56:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 3:56:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:56:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:56:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 15:56:57.301225 28664 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/rangerkms_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/rangerkms_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.254.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.254.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 15:56:57 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204617, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 15:56:57.463642 28664 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 15:56:57.485385 28664 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 15:56:58.292026 28664 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 15:56:58.292191 28664 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms
I20260808 15:56:58.292333 28664 mini_ranger_kms.cc:227] Using postgres at 127.27.254.20:59029
2026-08-08 15:56:59,112 [I] DB FLAVOR :POSTGRES
2026-08-08 15:56:59,112 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:56:59,112 [I] Checking connection
2026-08-08 15:56:59,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 15:56:59,355 [I] connection success
2026-08-08 15:56:59,355 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 15:56:59,356 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 15:56:59,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 15:56:59,607 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 15:56:59,608 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:56:59,608 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 15:56:59,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:59029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 15:56:59,852 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 15:57:00.319759 28664 mini_ranger_kms.cc:310] Created kms service
I20260808 15:57:00.486893 28664 mini_ranger_kms.cc:326] Created kudu user
I20260808 15:57:00.530776 28664 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 15:57:00 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204617, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:00.975819 28664 mini_ranger_kms.cc:384] Added ranger policy
I20260808 15:57:00.977567 28664 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 15:57:00.977651 28664 mini_ranger_kms.cc:240] Using host: 127.27.254.20
I20260808 15:57:00.996201 28664 mini_ranger_kms.cc:276] Ranger KMS PID: 7459
I20260808 15:57:00.996408 28664 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.254.20:48145
Aug 08, 2026 3:57:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:57:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:57:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:57:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:57:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:57:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 15:57:03.285408 28664 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.822s user 0.125s sys 0.399s
I20260808 15:57:03.285810 28664 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 15:57:03.286036 28664 mini_ranger_kms.cc:401] 127.27.254.20:48145/kms/v1/keys
Aug 08 15:57:03 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204623, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.254.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 15:57:03 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204623, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.254.20@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
Aug 08 15:57:04 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204623, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.254.20@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
Aug 08 15:57:06 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:57:06.486991 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:45511
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:45511
--ranger_config_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:48145/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:57:06.810328 7510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:06.810626 7510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:06.810781 7510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:06.817644 7510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:57:06.817740 7510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:06.817845 7510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:06.817950 7510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:57:06.818033 7510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:57:06.827653 7510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.254.20:48145/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:45511
--ranger_config_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:06.828948 7510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:06.830309 7510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:06.843613 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:06.843647 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:06.844277 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:06.844923 7510 server_base.cc:1034] running on GCE node
I20260808 15:57:06.845924 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:06.847898 7510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:06.849169 7510 hybrid_clock.cc:648] HybridClock initialized: now 1786204626849129 us; error 52 us; skew 500 ppm
I20260808 15:57:06.849669 7510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:06 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:06.853570 7510 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:57:06.855482 7510 webserver.cc:460] Webserver started at http://127.27.254.62:41805/ using document root <none> and password file <none>
I20260808 15:57:06.856350 7510 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:06.856489 7510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:06.856843 7510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:06 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:06.910883 7510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "2b200735311b47b0baea029a883c9d93"
format_stamp: "Formatted at 2026-08-08 15:57:06 on dist-test-slave-srgv"
server_key: "94a2092531d0a1920a6d778e3245ff4c"
server_key_iv: "00606fff85c0dfc7130ea7e4756d8afc"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:06.911731 7510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "2b200735311b47b0baea029a883c9d93"
format_stamp: "Formatted at 2026-08-08 15:57:06 on dist-test-slave-srgv"
server_key: "94a2092531d0a1920a6d778e3245ff4c"
server_key_iv: "00606fff85c0dfc7130ea7e4756d8afc"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:06.917142 7510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:57:06.933471 7525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:06.935068 7510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:57:06.935271 7510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "2b200735311b47b0baea029a883c9d93"
format_stamp: "Formatted at 2026-08-08 15:57:06 on dist-test-slave-srgv"
server_key: "94a2092531d0a1920a6d778e3245ff4c"
server_key_iv: "00606fff85c0dfc7130ea7e4756d8afc"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:06.935501 7510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:06.967712 7510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:06.971998 7510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:06.972304 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:06.992389 7510 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:45511
I20260808 15:57:06.992422 7576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:45511 every 8 connection(s)
I20260808 15:57:06.993808 7510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:57:06.998575 7577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:57:07.002079 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 7510
I20260808 15:57:07.002292 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:57:07.002959 28664 external_mini_cluster.cc:1428] Setting key be88230f1bfa8bb820475da4186fd566
I20260808 15:57:07.013231 7577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93: Bootstrap starting.
I20260808 15:57:07.017618 7577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93: Neither blocks nor log segments found. Creating new log.
I20260808 15:57:07.019083 7577 log.cc:824] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93: Log is configured to *not* fsync() on all Append() calls
I20260808 15:57:07.022815 7577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93: No bootstrap required, opened a new log
Aug 08 15:57:07 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:57:07.033972 7577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b200735311b47b0baea029a883c9d93" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45511 } }
I20260808 15:57:07.034389 7577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:57:07.034490 7577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b200735311b47b0baea029a883c9d93, State: Initialized, Role: FOLLOWER
I20260808 15:57:07.035180 7577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b200735311b47b0baea029a883c9d93" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45511 } }
I20260808 15:57:07.035637 7577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:57:07.035939 7577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:57:07.036185 7577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:57:07.038482 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:07.005217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.27.254.62:45511)
0808 15:57:07.017985 (+ 12768us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:07.018004 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:07.018464 (+ 460us) server_negotiation.cc:407] Connection header received
0808 15:57:07.020958 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:07.020979 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:07.021347 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:07.021791 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:07.023159 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:07.024492 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:07.025818 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:07.026497 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:07.026711 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:07.030746 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:07.030754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:07.030767 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:07.030800 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:07.033871 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:07.034666 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:07.034676 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:07.034684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:07.034886 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:07.035287 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:07.035293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:07.035296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:07.036075 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:07.036398 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:07.037025 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 15:57:07.037488 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12489,"thread_start_us":12180,"threads_started":1}
I20260808 15:57:07.039536 7577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b200735311b47b0baea029a883c9d93" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45511 } }
I20260808 15:57:07.041806 7577 leader_election.cc:302] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b200735311b47b0baea029a883c9d93; no voters:
I20260808 15:57:07.042459 7577 leader_election.cc:288] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:57:07.042567 7582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:57:07.044508 7577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:57:07.045727 7582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [term 1 LEADER]: Becoming Leader. State: Replica: 2b200735311b47b0baea029a883c9d93, State: Running, Role: LEADER
I20260808 15:57:07.046289 7582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b200735311b47b0baea029a883c9d93" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45511 } }
I20260808 15:57:07.055248 7583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b200735311b47b0baea029a883c9d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b200735311b47b0baea029a883c9d93" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45511 } } }
I20260808 15:57:07.055660 7583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [sys.catalog]: This master's current role is: LEADER
I20260808 15:57:07.056108 7584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b200735311b47b0baea029a883c9d93. Latest consensus state: current_term: 1 leader_uuid: "2b200735311b47b0baea029a883c9d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b200735311b47b0baea029a883c9d93" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 45511 } } }
I20260808 15:57:07.056422 7584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93 [sys.catalog]: This master's current role is: LEADER
I20260808 15:57:07.056623 7589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:57:07.072774 7589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:57:07.091729 7577 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:57:07.093194 7589 catalog_manager.cc:1347] Generated new cluster ID: c644a1e7df624fb997ddd51a3e770930
I20260808 15:57:07.093389 7589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:57:07.119495 7589 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:57:07.121546 7589 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:57:07.130584 7589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93: Generated new TSK 0
I20260808 15:57:07.131673 7589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:57:07.898280 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:48145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:45511
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:57:08 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:57:08.339939 7622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:08.340307 7622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:08.340467 7622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:08.351792 7622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:08.351941 7622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:08.352136 7622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:57:08.367924 7622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.254.20:48145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:08.369695 7622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:08.371486 7622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:08.389559 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:08.390590 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:08.397737 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:08.398125 7622 server_base.cc:1034] running on GCE node
I20260808 15:57:08.398795 7622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:08.399802 7622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:08.401042 7622 hybrid_clock.cc:648] HybridClock initialized: now 1786204628400984 us; error 49 us; skew 500 ppm
I20260808 15:57:08.401386 7622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:08 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:08.404739 7622 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:57:08.406411 7622 webserver.cc:460] Webserver started at http://127.27.254.1:33895/ using document root <none> and password file <none>
I20260808 15:57:08.407210 7622 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:08.407354 7622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:08.407691 7622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:08 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:08.427353 7622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "c353cb3024c94807b82bc3387beafa32"
format_stamp: "Formatted at 2026-08-08 15:57:08 on dist-test-slave-srgv"
server_key: "3c2e3a9e3589a02e8f058e7ea97609a5"
server_key_iv: "f41cbfe5d91178435a2aba328ee604ad"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:08.428546 7622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "c353cb3024c94807b82bc3387beafa32"
format_stamp: "Formatted at 2026-08-08 15:57:08 on dist-test-slave-srgv"
server_key: "3c2e3a9e3589a02e8f058e7ea97609a5"
server_key_iv: "f41cbfe5d91178435a2aba328ee604ad"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:08.438645 7622 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260808 15:57:08.468152 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:08.470176 7622 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:57:08.470422 7622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "c353cb3024c94807b82bc3387beafa32"
format_stamp: "Formatted at 2026-08-08 15:57:08 on dist-test-slave-srgv"
server_key: "3c2e3a9e3589a02e8f058e7ea97609a5"
server_key_iv: "f41cbfe5d91178435a2aba328ee604ad"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:08.470697 7622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:08.508854 7622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:08.514240 7622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:08.514601 7622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:08.515805 7622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:57:08.522274 7622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:57:08.522374 7622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:08.522505 7622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:57:08.522572 7622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:57:08 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:08.840087 7754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:46823 every 8 connection(s)
I20260808 15:57:08.840504 7622 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:46823
I20260808 15:57:08.842314 7622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:57:08.850061 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 7622
I20260808 15:57:08.850328 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:57:08.851330 28664 external_mini_cluster.cc:1428] Setting key 160410b41fa38a04a52fa454835c238f
Aug 08 15:57:08 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:57:08.907361 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:08.848321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:54295 (local address 127.27.254.62:45511)
0808 15:57:08.871206 (+ 22885us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:08.871213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:08.871235 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:57:08.871301 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:08.871306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:08.871398 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:08.871526 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:08.878509 (+ 6983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:08.879684 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:08.885548 (+ 5864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:08.886230 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:08.886348 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:08.894352 (+ 8004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:08.894358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:08.894363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:08.894419 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:08.898019 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:08.899055 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:08.899061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:08.899064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:08.899223 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:08.899851 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:08.899857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:08.899860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:08.900242 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:08.900387 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:08.906681 (+ 6294us) server_negotiation.cc:299] Negotiation successful
0808 15:57:08.906962 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22755,"thread_start_us":23301,"threads_started":1}
I20260808 15:57:08.910162 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:08.849046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:45511 (local address 127.27.254.1:54295)
0808 15:57:08.864139 (+ 15093us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:08.864199 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:08.866716 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:08.876504 (+ 9788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:08.876520 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:08.877259 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:57:08.878237 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:08.878268 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:08.879918 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:08.879937 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:08.882055 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:08.882071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:08.886744 (+ 4673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:08.886751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:08.887087 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:08.891057 (+ 3970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:57:08.891103 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:57:08.894061 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:57:08.898230 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:08.898240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:08.898261 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:08.898882 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:08.899379 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:08.899384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:08.899388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:08.899706 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:08.900485 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:57:08.900494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:57:08.900993 (+ 499us) client_negotiation.cc:769] Sending connection context
0808 15:57:08.907973 (+ 6980us) client_negotiation.cc:240] Negotiation successful
0808 15:57:08.908622 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14734,"thread_start_us":18022,"threads_started":1}
I20260808 15:57:08.912381 7755 heartbeater.cc:347] Connected to a master server at 127.27.254.62:45511
I20260808 15:57:08.912714 7755 heartbeater.cc:464] Registering TS with master...
I20260808 15:57:08.913657 7755 heartbeater.cc:511] Master 127.27.254.62:45511 requested a full tablet report, sending...
I20260808 15:57:08.916342 7542 ts_manager.cc:194] Registered new tserver with Master: c353cb3024c94807b82bc3387beafa32 (127.27.254.1:46823)
I20260808 15:57:08.918455 7542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:54295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
W20260808 15:57:08.972498 28664 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.254.2: real 0.118s user 0.001s sys 0.018s
I20260808 15:57:08.973610 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:48145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:45511
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:57:08 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204628, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
W20260808 15:57:09.447101 7762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:09.447391 7762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:09.447573 7762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:09.455209 7762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:09.455371 7762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:09.455557 7762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:57:09.465525 7762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.254.20:48145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:09.466854 7762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:09.468143 7762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:09.492976 7775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:09.498974 7776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:09.503539 7762 server_base.cc:1034] running on GCE node
W20260808 15:57:09.503827 7778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:09.504638 7762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:09.505900 7762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:09.507175 7762 hybrid_clock.cc:648] HybridClock initialized: now 1786204629507117 us; error 43 us; skew 500 ppm
I20260808 15:57:09.507548 7762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:09 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:09.511667 7762 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:57:09.513358 7762 webserver.cc:460] Webserver started at http://127.27.254.2:46411/ using document root <none> and password file <none>
I20260808 15:57:09.514104 7762 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:09.514232 7762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:09.514561 7762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:09 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:09.531607 7762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "96cbe49c94604523baf08d6fcd0d7abf"
format_stamp: "Formatted at 2026-08-08 15:57:09 on dist-test-slave-srgv"
server_key: "4fe631d446249765baca579c16ace723"
server_key_iv: "9207a39a764e714725423ad292d2d4df"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:09.532498 7762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "96cbe49c94604523baf08d6fcd0d7abf"
format_stamp: "Formatted at 2026-08-08 15:57:09 on dist-test-slave-srgv"
server_key: "4fe631d446249765baca579c16ace723"
server_key_iv: "9207a39a764e714725423ad292d2d4df"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:09.538501 7762 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 15:57:09.555670 7785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:09.557186 7762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:57:09.557345 7762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "96cbe49c94604523baf08d6fcd0d7abf"
format_stamp: "Formatted at 2026-08-08 15:57:09 on dist-test-slave-srgv"
server_key: "4fe631d446249765baca579c16ace723"
server_key_iv: "9207a39a764e714725423ad292d2d4df"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:09.557611 7762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:09.571014 7762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:09.574882 7762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:09.575179 7762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:09.576277 7762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:57:09.577785 7762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:57:09.577855 7762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:09.578027 7762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:57:09.578083 7762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:09.617736 7762 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:40549
I20260808 15:57:09.617764 7897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:40549 every 8 connection(s)
I20260808 15:57:09.619026 7762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:57:09.620625 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 7762
I20260808 15:57:09.620795 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:57:09.621456 28664 external_mini_cluster.cc:1428] Setting key 65cc1bfe6c0ebd4f90e07db63c86cd09
Aug 08 15:57:09 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204629, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:57:09.640805 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:09.621939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:38563 (local address 127.27.254.62:45511)
0808 15:57:09.622823 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:09.622830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:09.625481 (+ 2651us) server_negotiation.cc:407] Connection header received
0808 15:57:09.625676 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:09.625680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:09.625765 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:09.625884 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:09.627283 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:09.628033 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:09.629683 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:09.630108 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:09.630252 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:09.633575 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:09.633581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:09.633586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:09.633622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:09.636995 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:09.638011 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:09.638017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:09.638020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:09.638166 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:09.638699 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:09.638705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:09.638708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:09.639107 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:09.639265 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:09.640189 (+ 924us) server_negotiation.cc:299] Negotiation successful
0808 15:57:09.640418 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":615,"threads_started":1}
I20260808 15:57:09.642241 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:09.622731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:45511 (local address 127.27.254.2:38563)
0808 15:57:09.623594 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:09.623654 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:09.625375 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:09.626139 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:09.626146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:09.626555 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:57:09.627102 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:09.627119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:09.628191 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:09.628196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:09.629538 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:09.629551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:09.630339 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:09.630346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:09.630591 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:09.631280 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:57:09.631300 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:57:09.633347 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:57:09.637140 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:09.637152 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:09.637170 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:09.637873 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:09.638290 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:09.638298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:09.638305 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:09.638561 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:09.639257 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:57:09.639268 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:57:09.639732 (+ 464us) client_negotiation.cc:769] Sending connection context
0808 15:57:09.640072 (+ 340us) client_negotiation.cc:240] Negotiation successful
0808 15:57:09.640633 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":403,"threads_started":1}
I20260808 15:57:09.643810 7898 heartbeater.cc:347] Connected to a master server at 127.27.254.62:45511
I20260808 15:57:09.644150 7898 heartbeater.cc:464] Registering TS with master...
I20260808 15:57:09.645053 7898 heartbeater.cc:511] Master 127.27.254.62:45511 requested a full tablet report, sending...
I20260808 15:57:09.646816 7542 ts_manager.cc:194] Registered new tserver with Master: 96cbe49c94604523baf08d6fcd0d7abf (127.27.254.2:40549)
I20260808 15:57:09.647881 7542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:38563
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:57:09.681121 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:48145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:45511
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:57:09.922456 7755 heartbeater.cc:503] Master 127.27.254.62:45511 was elected leader, sending a full tablet report...
W20260808 15:57:10.019744 7904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:10.020079 7904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:10.020256 7904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:10.028232 7904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:10.028396 7904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:10.028580 7904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:57:10.039033 7904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.254.20:48145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:10.040479 7904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:10.041816 7904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:10.055004 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:10.055043 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:10.055076 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:10.056216 7904 server_base.cc:1034] running on GCE node
I20260808 15:57:10.056849 7904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:10.057780 7904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:10.058975 7904 hybrid_clock.cc:648] HybridClock initialized: now 1786204630058918 us; error 54 us; skew 500 ppm
I20260808 15:57:10.059293 7904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:10 dist-test-slave-srgv krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:10.062552 7904 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:57:10.064052 7904 webserver.cc:460] Webserver started at http://127.27.254.3:36791/ using document root <none> and password file <none>
I20260808 15:57:10.064764 7904 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:10.064910 7904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:10.065214 7904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:10 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:10.079537 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "726491e689374e7db687f93cfe04bf90"
format_stamp: "Formatted at 2026-08-08 15:57:10 on dist-test-slave-srgv"
server_key: "db3556554b2c668446a38d10b7196cde"
server_key_iv: "b21c6f2da0214f5875c0dbf8dbad3e4f"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:10.080397 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "726491e689374e7db687f93cfe04bf90"
format_stamp: "Formatted at 2026-08-08 15:57:10 on dist-test-slave-srgv"
server_key: "db3556554b2c668446a38d10b7196cde"
server_key_iv: "b21c6f2da0214f5875c0dbf8dbad3e4f"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:10.086073 7904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:57:10.100181 7919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:10.101743 7904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:57:10.101945 7904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "726491e689374e7db687f93cfe04bf90"
format_stamp: "Formatted at 2026-08-08 15:57:10 on dist-test-slave-srgv"
server_key: "db3556554b2c668446a38d10b7196cde"
server_key_iv: "b21c6f2da0214f5875c0dbf8dbad3e4f"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:10.102185 7904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:10.146595 7904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:10.150487 7904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:10.150789 7904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:10.151717 7904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:57:10.153107 7904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:57:10.153208 7904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:10.153355 7904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:57:10.153445 7904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:10.188494 7904 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:46213
I20260808 15:57:10.188530 8031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:46213 every 8 connection(s)
I20260808 15:57:10.189904 7904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:57:10.192986 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 7904
I20260808 15:57:10.193177 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:57:10.193827 28664 external_mini_cluster.cc:1428] Setting key f11f7c7f61064cae6c89a73a9d3346f4
Aug 08 15:57:10 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:57:10.209882 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.192406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:50983 (local address 127.27.254.62:45511)
0808 15:57:10.192945 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.192952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.194126 (+ 1174us) server_negotiation.cc:407] Connection header received
0808 15:57:10.196472 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.196477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.196565 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.196706 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:10.198224 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.198949 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.200395 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.200791 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.200887 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.204065 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:10.204069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:10.204072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:10.204101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:10.206708 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.207484 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.207489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.207492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.207645 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.208159 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.208163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.208165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.208519 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:10.208664 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.209399 (+ 735us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.209605 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":231,"threads_started":1}
I20260808 15:57:10.210724 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.192983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:45511 (local address 127.27.254.3:50983)
0808 15:57:10.193929 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:10.193995 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:10.196229 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:10.196959 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:10.196973 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:10.197499 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:57:10.198083 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.198100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.199076 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.199080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.200228 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.200238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.200996 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.201000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.201193 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.201870 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:57:10.201890 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:57:10.203850 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:57:10.206809 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:10.206814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:10.206823 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:10.207350 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:10.207769 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:10.207774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:10.207776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:10.208036 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:10.208652 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:57:10.208659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:57:10.209019 (+ 360us) client_negotiation.cc:769] Sending connection context
0808 15:57:10.209298 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 15:57:10.209692 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":385,"threads_started":1}
I20260808 15:57:10.212034 8032 heartbeater.cc:347] Connected to a master server at 127.27.254.62:45511
I20260808 15:57:10.212303 8032 heartbeater.cc:464] Registering TS with master...
I20260808 15:57:10.212967 8032 heartbeater.cc:511] Master 127.27.254.62:45511 requested a full tablet report, sending...
I20260808 15:57:10.214316 7542 ts_manager.cc:194] Registered new tserver with Master: 726491e689374e7db687f93cfe04bf90 (127.27.254.3:46213)
I20260808 15:57:10.215116 7542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:50983
I20260808 15:57:10.221052 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:57:10.238101 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.225577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.27.254.62:45511)
0808 15:57:10.225726 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.225731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.226237 (+ 506us) server_negotiation.cc:407] Connection header received
0808 15:57:10.226425 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.226433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.226511 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.226615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:10.227952 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.228636 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.230165 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.230656 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.230793 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.233083 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:10.233089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:10.233093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:10.233129 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:10.235479 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.236163 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.236169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.236172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.236294 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.236683 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.236690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.236694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.236957 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:10.237211 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.237565 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.237777 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:57:10.241930 7542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:57:10.249533 7542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:57:10.271386 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.261364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.27.254.2:40549)
0808 15:57:10.263058 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.263066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.263092 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:57:10.263182 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.263192 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.263480 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.264440 (+ 960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:57:10.264672 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.266121 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.268865 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.270216 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.270370 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.270594 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.270872 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.271068 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1344,"threads_started":1}
I20260808 15:57:10.271423 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.261212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40549 (local address 127.0.0.1:33316)
0808 15:57:10.262293 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:10.262346 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:10.262472 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:10.263770 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:10.263781 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:10.263821 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:57:10.264215 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.264231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.266248 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.266256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.268682 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.268703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.270433 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.270439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.270635 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.270686 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 15:57:10.270833 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 15:57:10.271025 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":593,"threads_started":1}
I20260808 15:57:10.272163 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.262787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:46213 (local address 127.0.0.1:38298)
0808 15:57:10.263433 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:10.263449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:10.263554 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:10.264769 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:10.264777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:10.264809 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:57:10.265265 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.265279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.267557 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.267562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.269488 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.269503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.271348 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.271353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.271564 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.271593 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:57:10.271671 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 15:57:10.271793 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":19,"thread_start_us":400,"threads_started":1}
I20260808 15:57:10.272496 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.263145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.27.254.3:46213)
0808 15:57:10.264141 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.264150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.264182 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:57:10.264275 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.264283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.264537 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.265500 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:57:10.265833 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.267399 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.269644 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.271126 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.271503 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.271716 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.271775 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.271927 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":651,"threads_started":1}
I20260808 15:57:10.275938 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.263494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:46823 (local address 127.0.0.1:55878)
0808 15:57:10.266742 (+ 3248us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:10.266759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:10.266875 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:10.267430 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:10.267436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:10.267468 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:57:10.267945 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.267956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.269838 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.269844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.273139 (+ 3295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.273154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.275051 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.275057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.275258 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.275289 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:57:10.275394 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 15:57:10.275529 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3124,"mutex_wait_us":3,"thread_start_us":2952,"threads_started":1}
I20260808 15:57:10.275933 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.264091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.27.254.1:46823)
0808 15:57:10.265757 (+ 1666us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.265765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.266869 (+ 1104us) server_negotiation.cc:407] Connection header received
0808 15:57:10.267024 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.267031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.267233 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.267430 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:57:10.268153 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.269699 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.273316 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.274847 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.274984 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.275198 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.275439 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.275595 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":1350,"threads_started":1}
I20260808 15:57:10.275447 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 8aba79a8eb284bc6b4e6b6455bf5513e (DEFAULT_TABLE table=test-table [id=ece0d7d9e5974636932ac4c0cf76b2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:57:10.276000 7967 tablet_service.cc:1467] Processing CreateTablet for tablet 8aba79a8eb284bc6b4e6b6455bf5513e (DEFAULT_TABLE table=test-table [id=ece0d7d9e5974636932ac4c0cf76b2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:57:10.276885 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aba79a8eb284bc6b4e6b6455bf5513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:57:10.277474 7967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aba79a8eb284bc6b4e6b6455bf5513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:57:10.279496 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 8aba79a8eb284bc6b4e6b6455bf5513e (DEFAULT_TABLE table=test-table [id=ece0d7d9e5974636932ac4c0cf76b2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:57:10.281536 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aba79a8eb284bc6b4e6b6455bf5513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:57:10.288793 8051 tablet_bootstrap.cc:492] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Bootstrap starting.
I20260808 15:57:10.291723 8052 tablet_bootstrap.cc:492] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Bootstrap starting.
I20260808 15:57:10.292241 8051 tablet_bootstrap.cc:654] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Neither blocks nor log segments found. Creating new log.
I20260808 15:57:10.293623 8051 log.cc:824] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Log is configured to *not* fsync() on all Append() calls
I20260808 15:57:10.295639 8052 tablet_bootstrap.cc:654] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Neither blocks nor log segments found. Creating new log.
I20260808 15:57:10.295841 8054 tablet_bootstrap.cc:492] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Bootstrap starting.
I20260808 15:57:10.297312 8052 log.cc:824] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Log is configured to *not* fsync() on all Append() calls
I20260808 15:57:10.298874 8054 tablet_bootstrap.cc:654] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Neither blocks nor log segments found. Creating new log.
I20260808 15:57:10.299975 8051 tablet_bootstrap.cc:492] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: No bootstrap required, opened a new log
I20260808 15:57:10.300096 8054 log.cc:824] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Log is configured to *not* fsync() on all Append() calls
I20260808 15:57:10.300302 8051 ts_tablet_manager.cc:1397] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260808 15:57:10.300549 8052 tablet_bootstrap.cc:492] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: No bootstrap required, opened a new log
I20260808 15:57:10.300873 8052 ts_tablet_manager.cc:1397] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:57:10.303092 8054 tablet_bootstrap.cc:492] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: No bootstrap required, opened a new log
I20260808 15:57:10.303421 8054 ts_tablet_manager.cc:1397] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:57:10.308346 8051 raft_consensus.cc:348] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.308701 8051 raft_consensus.cc:374] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:57:10.308806 8051 raft_consensus.cc:729] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726491e689374e7db687f93cfe04bf90, State: Initialized, Role: FOLLOWER
I20260808 15:57:10.309454 8051 consensus_queue.cc:260] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.310526 8032 heartbeater.cc:503] Master 127.27.254.62:45511 was elected leader, sending a full tablet report...
I20260808 15:57:10.311187 8051 ts_tablet_manager.cc:1428] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260808 15:57:10.311298 8052 raft_consensus.cc:348] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.311667 8052 raft_consensus.cc:374] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:57:10.311815 8052 raft_consensus.cc:729] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbe49c94604523baf08d6fcd0d7abf, State: Initialized, Role: FOLLOWER
I20260808 15:57:10.312378 8052 consensus_queue.cc:260] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.314142 7898 heartbeater.cc:503] Master 127.27.254.62:45511 was elected leader, sending a full tablet report...
I20260808 15:57:10.314412 8052 ts_tablet_manager.cc:1428] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260808 15:57:10.314307 8054 raft_consensus.cc:348] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.314702 8054 raft_consensus.cc:374] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:57:10.314766 8054 raft_consensus.cc:729] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c353cb3024c94807b82bc3387beafa32, State: Initialized, Role: FOLLOWER
I20260808 15:57:10.315243 8054 consensus_queue.cc:260] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.316962 8054 ts_tablet_manager.cc:1428] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260808 15:57:10.363907 7758 tablet.cc:2369] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:57:10.373183 7899 tablet.cc:2369] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:57:10.436605 8058 raft_consensus.cc:482] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:57:10.436909 8058 raft_consensus.cc:504] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.438764 8058 leader_election.cc:288] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 726491e689374e7db687f93cfe04bf90 (127.27.254.3:46213), c353cb3024c94807b82bc3387beafa32 (127.27.254.1:46823)
W20260808 15:57:10.441584 8033 tablet.cc:2369] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:57:10.446684 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.439219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:33417 (local address 127.27.254.3:46213)
0808 15:57:10.439431 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.439437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.439790 (+ 353us) server_negotiation.cc:407] Connection header received
0808 15:57:10.440064 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.440073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.440171 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.440255 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:57:10.441025 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.441965 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.444586 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.445842 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.446033 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.446109 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.446358 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.446486 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:57:10.446930 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.438954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:46213 (local address 127.27.254.2:33417)
0808 15:57:10.439679 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:10.439697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:10.439839 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:10.440480 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:10.440483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:10.440504 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:57:10.440885 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.440894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.442100 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.442104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.444416 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.444435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.446054 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.446058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.446173 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.446196 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:57:10.446323 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 15:57:10.446482 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":22,"spinlock_wait_cycles":26368,"thread_start_us":386,"threads_started":1}
I20260808 15:57:10.447072 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.440063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:33329 (local address 127.27.254.1:46823)
0808 15:57:10.440292 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.440297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.440575 (+ 278us) server_negotiation.cc:407] Connection header received
0808 15:57:10.440803 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.440808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.440876 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.440978 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:57:10.441691 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.442571 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.444918 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.446170 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.446308 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.446372 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.446723 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.446852 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 15:57:10.447220 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.439761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:46823 (local address 127.27.254.2:33329)
0808 15:57:10.440429 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:10.440473 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:10.440653 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:10.440983 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:10.440987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:10.441009 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:57:10.441501 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.441516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.442731 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.442742 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.444789 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:10.444798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.446362 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.446368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:10.446550 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.446592 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 15:57:10.446678 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 15:57:10.446787 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":24,"thread_start_us":342,"threads_started":1}
I20260808 15:57:10.448042 7987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aba79a8eb284bc6b4e6b6455bf5513e" candidate_uuid: "96cbe49c94604523baf08d6fcd0d7abf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "726491e689374e7db687f93cfe04bf90" is_pre_election: true
I20260808 15:57:10.448217 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aba79a8eb284bc6b4e6b6455bf5513e" candidate_uuid: "96cbe49c94604523baf08d6fcd0d7abf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c353cb3024c94807b82bc3387beafa32" is_pre_election: true
I20260808 15:57:10.448534 7987 raft_consensus.cc:2454] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbe49c94604523baf08d6fcd0d7abf in term 0.
I20260808 15:57:10.448741 7710 raft_consensus.cc:2454] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbe49c94604523baf08d6fcd0d7abf in term 0.
I20260808 15:57:10.449184 7789 leader_election.cc:302] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 726491e689374e7db687f93cfe04bf90, 96cbe49c94604523baf08d6fcd0d7abf; no voters:
I20260808 15:57:10.449585 8058 raft_consensus.cc:2781] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:57:10.449730 8058 raft_consensus.cc:482] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:57:10.449844 8058 raft_consensus.cc:3037] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:57:10.452081 8058 raft_consensus.cc:504] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.452843 8058 leader_election.cc:288] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [CANDIDATE]: Term 1 election: Requested vote from peers 726491e689374e7db687f93cfe04bf90 (127.27.254.3:46213), c353cb3024c94807b82bc3387beafa32 (127.27.254.1:46823)
I20260808 15:57:10.453212 7987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aba79a8eb284bc6b4e6b6455bf5513e" candidate_uuid: "96cbe49c94604523baf08d6fcd0d7abf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "726491e689374e7db687f93cfe04bf90"
I20260808 15:57:10.453284 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aba79a8eb284bc6b4e6b6455bf5513e" candidate_uuid: "96cbe49c94604523baf08d6fcd0d7abf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c353cb3024c94807b82bc3387beafa32"
I20260808 15:57:10.453435 7987 raft_consensus.cc:3037] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:57:10.453536 7710 raft_consensus.cc:3037] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:57:10.455590 7987 raft_consensus.cc:2454] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbe49c94604523baf08d6fcd0d7abf in term 1.
I20260808 15:57:10.455756 7710 raft_consensus.cc:2454] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbe49c94604523baf08d6fcd0d7abf in term 1.
I20260808 15:57:10.455963 7789 leader_election.cc:302] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 726491e689374e7db687f93cfe04bf90, 96cbe49c94604523baf08d6fcd0d7abf; no voters:
I20260808 15:57:10.456214 8058 raft_consensus.cc:2781] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:57:10.456633 8058 raft_consensus.cc:686] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 1 LEADER]: Becoming Leader. State: Replica: 96cbe49c94604523baf08d6fcd0d7abf, State: Running, Role: LEADER
I20260808 15:57:10.457108 8058 consensus_queue.cc:237] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } }
I20260808 15:57:10.461790 7542 catalog_manager.cc:5495] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cbe49c94604523baf08d6fcd0d7abf (127.27.254.2). New cstate: current_term: 1 leader_uuid: "96cbe49c94604523baf08d6fcd0d7abf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96cbe49c94604523baf08d6fcd0d7abf" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 } health_report { overall_health: UNKNOWN } } }
I20260808 15:57:10.511179 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.506111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.27.254.2:40549)
0808 15:57:10.506299 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.506303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.506320 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:57:10.506529 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.506532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.506607 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.506726 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:57:10.507021 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.507709 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.509103 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.509542 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.509633 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.510066 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:57:10.510574 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:57:10.510680 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.510808 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.510869 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 15:57:10.519137 7710 raft_consensus.cc:1261] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 1 FOLLOWER]: Refusing update from remote peer 96cbe49c94604523baf08d6fcd0d7abf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:57:10.519137 7987 raft_consensus.cc:1261] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 1 FOLLOWER]: Refusing update from remote peer 96cbe49c94604523baf08d6fcd0d7abf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:57:10.520154 8062 consensus_queue.cc:1035] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [LEADER]: Connected to new peer: Peer: permanent_uuid: "726491e689374e7db687f93cfe04bf90" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:57:10.520519 8058 consensus_queue.cc:1035] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c353cb3024c94807b82bc3387beafa32" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:57:10.536057 8066 mvcc.cc:205] Tried to move back new op lower bound from 7316294166596014080 to 7316294166356545536. Current Snapshot: MvccSnapshot[applied={T|T < 7316294166596014080}]
I20260808 15:57:10.544879 7541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:57:10.545192 7541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:57:10.548890 7541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b200735311b47b0baea029a883c9d93: Sending DeleteTablet for 3 replicas of tablet 8aba79a8eb284bc6b4e6b6455bf5513e
I20260808 15:57:10.550117 7833 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aba79a8eb284bc6b4e6b6455bf5513e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:57:10 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33316
I20260808 15:57:10.550351 7967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aba79a8eb284bc6b4e6b6455bf5513e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:57:10 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:38298
I20260808 15:57:10.550586 7690 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aba79a8eb284bc6b4e6b6455bf5513e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:57:10 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:55878
I20260808 15:57:10.553344 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 7622
I20260808 15:57:10.561918 8075 tablet_replica.cc:330] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: stopping tablet replica
I20260808 15:57:10.562322 8075 raft_consensus.cc:2229] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:57:10.562855 8075 raft_consensus.cc:2258] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:57:10.565171 8075 ts_tablet_manager.cc:1905] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:57:10.566303 8076 tablet_replica.cc:330] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: stopping tablet replica
I20260808 15:57:10.566792 8076 raft_consensus.cc:2229] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:57:10.567513 8076 raft_consensus.cc:2258] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:57:10.569357 8077 tablet_replica.cc:330] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: stopping tablet replica
I20260808 15:57:10.569802 8077 raft_consensus.cc:2229] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:57:10.570323 8077 raft_consensus.cc:2258] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:57:10.570704 8076 ts_tablet_manager.cc:1905] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:57:10.573379 8075 ts_tablet_manager.cc:1918] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:57:10.573637 8075 log.cc:1196] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/8aba79a8eb284bc6b4e6b6455bf5513e
I20260808 15:57:10.574173 8075 ts_tablet_manager.cc:1939] T 8aba79a8eb284bc6b4e6b6455bf5513e P 726491e689374e7db687f93cfe04bf90: Deleting consensus metadata
I20260808 15:57:10.576141 7529 catalog_manager.cc:4841] TS 726491e689374e7db687f93cfe04bf90 (127.27.254.3:46213): tablet 8aba79a8eb284bc6b4e6b6455bf5513e (table test-table [id=ece0d7d9e5974636932ac4c0cf76b2ea]) successfully deleted
I20260808 15:57:10.577998 8077 ts_tablet_manager.cc:1905] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:57:10.578537 8076 ts_tablet_manager.cc:1918] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:57:10.578718 8076 log.cc:1196] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/8aba79a8eb284bc6b4e6b6455bf5513e
I20260808 15:57:10.579242 8076 ts_tablet_manager.cc:1939] T 8aba79a8eb284bc6b4e6b6455bf5513e P 96cbe49c94604523baf08d6fcd0d7abf: Deleting consensus metadata
I20260808 15:57:10.580875 7529 catalog_manager.cc:4841] TS 96cbe49c94604523baf08d6fcd0d7abf (127.27.254.2:40549): tablet 8aba79a8eb284bc6b4e6b6455bf5513e (table test-table [id=ece0d7d9e5974636932ac4c0cf76b2ea]) successfully deleted
I20260808 15:57:10.584694 8077 ts_tablet_manager.cc:1918] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 08 15:57:10 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:57:10.584889 8077 log.cc:1196] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/8aba79a8eb284bc6b4e6b6455bf5513e
I20260808 15:57:10.585361 8077 ts_tablet_manager.cc:1939] T 8aba79a8eb284bc6b4e6b6455bf5513e P c353cb3024c94807b82bc3387beafa32: Deleting consensus metadata
I20260808 15:57:10.587597 7529 catalog_manager.cc:4841] TS c353cb3024c94807b82bc3387beafa32 (127.27.254.1:46823): tablet 8aba79a8eb284bc6b4e6b6455bf5513e (table test-table [id=ece0d7d9e5974636932ac4c0cf76b2ea]) successfully deleted
I20260808 15:57:10.592420 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.555195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.27.254.1:46823)
0808 15:57:10.555444 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.555453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.568657 (+ 13204us) server_negotiation.cc:407] Connection header received
0808 15:57:10.568866 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.568877 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.568992 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.569111 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:10.572373 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.573548 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.575818 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.576529 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.576631 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.586169 (+ 9538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:10.586182 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:10.586199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:10.586240 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:10.589393 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.590331 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.590338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.590340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.590471 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.590932 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.590935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.590937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.591273 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:10.591478 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.591939 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.592119 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:57:10.593029 7750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:10.684791 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 7622
I20260808 15:57:10.702888 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 7762
Aug 08 15:57:10 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:57:10.719070 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.703703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.27.254.2:40549)
0808 15:57:10.703887 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.703894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.703914 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:57:10.704083 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.704089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.704170 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.704295 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:10.705542 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.706434 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.707838 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.708267 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.708429 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.713136 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:10.713150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:10.713165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:10.713201 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:10.716229 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.716905 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.716918 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.716920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.717058 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.717546 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.717550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.717552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.717925 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:10.718266 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.718499 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.718717 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:57:10.719767 7893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:10.834220 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 7762
I20260808 15:57:10.848511 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 7904
Aug 08 15:57:10 dist-test-slave-srgv krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:57:10.863540 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:10.849234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.27.254.3:46213)
0808 15:57:10.849427 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:10.849434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:10.849483 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:57:10.849704 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:10.849708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:10.849770 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:10.849887 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:10.851311 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.852030 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.853271 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:10.853817 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:10.853961 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:10.858200 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:10.858213 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:10.858229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:10.858264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:10.860757 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.861447 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.861459 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.861467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.861593 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:10.861980 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:10.861985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:10.861988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:10.862343 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:10.862633 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:10.862948 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 15:57:10.863194 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:57:10.864166 8027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:10.952986 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 7904
I20260808 15:57:10.968883 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 7510
I20260808 15:57:10.970068 7572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:11.080169 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 7510
I20260808 15:57:11.101128 28664 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 15:57:11.436107 28664 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 15:57:11.436385 28664 mini_ranger.cc:66] Stopping Ranger...
I20260808 15:57:11.781389 28664 mini_ranger.cc:68] Stopped Ranger
2026-08-08 15:57:11.781 UTC [7094] LOG: received smart shutdown request
2026-08-08 15:57:11.785 UTC [7094] LOG: background worker "logical replication launcher" (PID 7103) exited with exit code 1
2026-08-08 15:57:11.785 UTC [7098] LOG: shutting down
2026-08-08 15:57:11.929 UTC [7094] LOG: database system is shut down
2026-08-08T15:57:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49755 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:57:12 dist-test-slave-srgv krb5kdc[8091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:57:12 dist-test-slave-srgv krb5kdc[8091](info): set up 2 sockets
Aug 08 15:57:12 dist-test-slave-srgv krb5kdc[8091](info): commencing operation
krb5kdc: starting...
W20260808 15:57:13.155776 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:57:13 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:57:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/rangeradmin_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/rangeradmin_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/rangerlookup_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/rangerlookup_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
I20260808 15:57:13.465921 28664 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBr1Jv8/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/postgres -l logfile start
2026-08-08 15:57:15.783 UTC [8121] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 15:57:15.783 UTC [8121] LOG: listening on IPv4 address "127.27.254.20", port 43675
2026-08-08 15:57:15.787 UTC [8121] LOG: listening on Unix socket "/tmp/.s.PGSQL.43675"
2026-08-08 15:57:15.809 UTC [8124] LOG: database system was shut down at 2026-08-08 15:57:14 UTC
2026-08-08 15:57:15.812 UTC [8121] LOG: database system is ready to accept connections
I20260808 15:57:16.779042 28664 mini_postgres.cc:96] Postgres bound to 43675
2026-08-08 15:57:16.796 UTC [8132] FATAL: database "slave" does not exist
127.27.254.20:43675 - accepting connections
I20260808 15:57:16.796955 28664 mini_ranger.cc:161] Starting Ranger...
I20260808 15:57:16.822440 28664 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 15:57:17.254326 28664 mini_ranger.cc:87] Created ranger Postgres database
I20260808 15:57:17.254441 28664 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-admin
2026-08-08 15:57:17,546 [I] DB FLAVOR :POSTGRES
2026-08-08 15:57:17,546 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:57:17,546 [I] Checking connection..
2026-08-08 15:57:17,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 15:57:17,780 [I] Checking connection passed.
2026-08-08 15:57:17,780 [I] --------- Verifying version history table ---------
2026-08-08 15:57:17,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:57:18,023 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:57:18,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:57:18,277 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:57:18,277 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 15:57:18,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 15:57:18.515 UTC [8196] WARNING: there is no transaction in progress
2026-08-08 15:57:18,521 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 15:57:18,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:57:18,777 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 15:57:18,778 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:57:18,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 15:57:19,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 15:57:19,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srgv.c.gcp-upstream.internal','N') ;"
2026-08-08 15:57:19,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:57:19,724 [I] Table x_portal_user does not exist in database ranger
2026-08-08 15:57:19,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:57:19,980 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 15:57:19,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:57:20,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:57:20,479 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 15:57:20,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 15:57:21.366 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.382 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.397 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.412 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.427 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.443 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.501 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.509 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.523 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.533 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.543 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.557 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.566 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.571 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.578 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:21.584 UTC [8304] WARNING: there is no transaction in progress
2026-08-08 15:57:22,172 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 15:57:22,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 15:57:22,421 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 15:57:22,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:57:22,691 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 15:57:22,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:57:22,922 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 15:57:22,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:57:23,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:57:23,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srgv.c.gcp-upstream.internal';"
2026-08-08 15:57:23,663 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 15:57:23,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 15:57:23,884 [I] DB_PATCHES have already been applied
I20260808 15:57:23.890110 28664 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBr1Jv8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 15:57:23.890193 28664 mini_ranger.cc:193] Using host: 127.27.254.20
I20260808 15:57:23.905519 28664 mini_ranger.cc:227] Ranger admin URL: http://127.27.254.20:44625
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:57:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 15:57:25.062368 28664 mini_ranger.cc:160] Time spent starting Ranger: real 8.265s user 0.002s sys 0.037s
Aug 08, 2026 3:57:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:57:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 3:57:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:57:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:57:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 15:57:46.652316 28664 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/rangerkms_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/rangerkms_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.254.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.254.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.254.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/HTTP_127.27.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 15:57:46 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204666, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 15:57:46.819685 28664 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 15:57:46.841341 28664 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 15:57:47.630137 28664 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 15:57:47.630260 28664 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms
I20260808 15:57:47.630317 28664 mini_ranger_kms.cc:227] Using postgres at 127.27.254.20:43675
2026-08-08 15:57:47,993 [I] DB FLAVOR :POSTGRES
2026-08-08 15:57:47,993 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:57:47,994 [I] Checking connection
2026-08-08 15:57:47,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 15:57:48,253 [I] connection success
2026-08-08 15:57:48,253 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 15:57:48,253 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 15:57:48,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 15:57:48,500 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 15:57:48,501 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:57:48,501 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 15:57:48,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.254.20:43675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 15:57:48,761 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 15:57:49.227229 28664 mini_ranger_kms.cc:310] Created kms service
I20260808 15:57:49.390202 28664 mini_ranger_kms.cc:326] Created kudu user
I20260808 15:57:49.434267 28664 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 15:57:49 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204666, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:49.806137 28664 mini_ranger_kms.cc:384] Added ranger policy
I20260808 15:57:49.806479 28664 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBr1Jv8/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBr1Jv8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 15:57:49.806574 28664 mini_ranger_kms.cc:240] Using host: 127.27.254.20
I20260808 15:57:49.822707 28664 mini_ranger_kms.cc:276] Ranger KMS PID: 8484
I20260808 15:57:49.822841 28664 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.254.20:40397
Aug 08, 2026 3:57:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:57:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:57:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:57:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 15:57:52 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204672, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.254.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:52.080693 28664 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.261s user 0.065s sys 0.315s
I20260808 15:57:52.080938 28664 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 15:57:52.081120 28664 mini_ranger_kms.cc:401] 127.27.254.20:40397/kms/v1/keys
Aug 08 15:57:52 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204672, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.254.20@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
Aug 08 15:57:52 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204672, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.254.20@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
Aug 08 15:57:55 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:57:55.136744 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41001
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:41001
--ranger_config_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:57:55.470121 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:55.470414 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:55.470579 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:55.477802 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:57:55.477921 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:55.478018 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:55.478117 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:57:55.478193 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:57:55.488247 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.254.20:40397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:41001
--ranger_config_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:55.489660 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:55.491009 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:55.503116 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:55.503077 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:55.503057 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:55.503437 8535 server_base.cc:1034] running on GCE node
I20260808 15:57:55.504230 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:55.505671 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:55.506889 8535 hybrid_clock.cc:648] HybridClock initialized: now 1786204675506843 us; error 36 us; skew 500 ppm
I20260808 15:57:55.507225 8535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:55 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:55.510217 8535 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:57:55.511658 8535 webserver.cc:460] Webserver started at http://127.27.254.62:39365/ using document root <none> and password file <none>
I20260808 15:57:55.512269 8535 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:55.512362 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:55.512624 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:55 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:55.578109 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "5a47501ba7cf4764b7d084999b39f8b8"
format_stamp: "Formatted at 2026-08-08 15:57:55 on dist-test-slave-srgv"
server_key: "5e910220ce3bdb43ff5083d8ba710f81"
server_key_iv: "a6024411aa7d056bfcce070db14d843a"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:55.578996 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "5a47501ba7cf4764b7d084999b39f8b8"
format_stamp: "Formatted at 2026-08-08 15:57:55 on dist-test-slave-srgv"
server_key: "5e910220ce3bdb43ff5083d8ba710f81"
server_key_iv: "a6024411aa7d056bfcce070db14d843a"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:55.584609 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:57:55.604660 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:55.606289 8535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:57:55.606491 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "5a47501ba7cf4764b7d084999b39f8b8"
format_stamp: "Formatted at 2026-08-08 15:57:55 on dist-test-slave-srgv"
server_key: "5e910220ce3bdb43ff5083d8ba710f81"
server_key_iv: "a6024411aa7d056bfcce070db14d843a"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:55.606726 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:55.624213 8535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:55.628366 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:55.628710 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:55.648461 8535 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:41001
I20260808 15:57:55.648483 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:41001 every 8 connection(s)
I20260808 15:57:55.649937 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:57:55.652377 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 8535
I20260808 15:57:55.652606 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:57:55.653543 28664 external_mini_cluster.cc:1428] Setting key 74bb280ae411f169d57aa9f2905b25ab
I20260808 15:57:55.654645 8602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:57:55 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:57:55.667104 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Bootstrap starting.
I20260808 15:57:55.670670 8602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Neither blocks nor log segments found. Creating new log.
I20260808 15:57:55.671742 8602 log.cc:824] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:57:55.674793 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: No bootstrap required, opened a new log
I20260808 15:57:55.675226 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:55.655762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.27.254.62:41001)
0808 15:57:55.656513 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:55.656525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:55.656566 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:57:55.658154 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:55.658175 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:55.658580 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:55.659002 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:55.660378 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:55.661822 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:55.663115 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:55.663824 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:55.664042 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:55.668006 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:55.668015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:55.668030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:55.668067 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:55.671158 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:55.671864 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:55.671873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:55.671883 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:55.672057 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:55.672444 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:55.672449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:55.672453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:55.672992 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:55.673311 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:55.673785 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 15:57:55.674195 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":342,"threads_started":1}
I20260808 15:57:55.682940 8602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:55.683323 8602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:57:55.683444 8602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a47501ba7cf4764b7d084999b39f8b8, State: Initialized, Role: FOLLOWER
I20260808 15:57:55.684062 8602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:55.684289 8602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:57:55.684425 8602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:57:55.684603 8602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:57:55.686838 8602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:55.687320 8602 leader_election.cc:302] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a47501ba7cf4764b7d084999b39f8b8; no voters:
I20260808 15:57:55.687927 8602 leader_election.cc:288] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:57:55.688057 8607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:57:55.688763 8607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5a47501ba7cf4764b7d084999b39f8b8, State: Running, Role: LEADER
I20260808 15:57:55.689242 8607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:55.690037 8602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:57:55.692013 8609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a47501ba7cf4764b7d084999b39f8b8. Latest consensus state: current_term: 1 leader_uuid: "5a47501ba7cf4764b7d084999b39f8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } } }
I20260808 15:57:55.692366 8609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:57:55.692768 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a47501ba7cf4764b7d084999b39f8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } } }
I20260808 15:57:55.693084 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:57:55.693673 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:57:55.699501 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:57:55.700248 8602 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:57:55.717011 8614 catalog_manager.cc:1347] Generated new cluster ID: 368d7f858cf445a6b2231a5c7fa5e67e
I20260808 15:57:55.717199 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:57:55.741756 8614 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:57:55.743615 8614 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:57:55.756435 8614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Generated new TSK 0
I20260808 15:57:55.757380 8614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:57:56.501680 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:41001
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:57:56 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:57:56.913573 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:56.914009 8647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:56.914191 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:56.925736 8647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:56.925895 8647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:56.926086 8647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:57:56.941813 8647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.254.20:40397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:56.943498 8647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:56.945235 8647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:56.962201 8657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:56.962756 8658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:56.971525 8660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:56.971796 8647 server_base.cc:1034] running on GCE node
I20260808 15:57:56.972810 8647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:56.974232 8647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:56.975553 8647 hybrid_clock.cc:648] HybridClock initialized: now 1786204676975486 us; error 49 us; skew 500 ppm
I20260808 15:57:56.975962 8647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:56 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:56.980422 8647 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:57:56.993824 8647 webserver.cc:460] Webserver started at http://127.27.254.1:40207/ using document root <none> and password file <none>
I20260808 15:57:56.994979 8647 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:56.995184 8647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:56.995625 8647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:56 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:57.015640 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "d01a69babd4f411989b3ee4cdf1ad438"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "e9069e4387a23a8a7711fa44ad66290a"
server_key_iv: "6ed184e4c4d9e63d6d326fe03fcc0048"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:57.016997 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "d01a69babd4f411989b3ee4cdf1ad438"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "e9069e4387a23a8a7711fa44ad66290a"
server_key_iv: "6ed184e4c4d9e63d6d326fe03fcc0048"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:57.024277 8647 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260808 15:57:57.041992 8667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:57.043599 8647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 15:57:57.043792 8647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "d01a69babd4f411989b3ee4cdf1ad438"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "e9069e4387a23a8a7711fa44ad66290a"
server_key_iv: "6ed184e4c4d9e63d6d326fe03fcc0048"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:57.044036 8647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:57.100852 8647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:57.106315 8647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:57.106734 8647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:57.110713 8647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:57:57.118302 8647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:57:57.118417 8647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:57.118595 8647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:57:57.118664 8647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:57.175344 8647 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:38325
I20260808 15:57:57.177253 8647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:57:57.175375 8779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:38325 every 8 connection(s)
I20260808 15:57:57.183763 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 8647
I20260808 15:57:57.183985 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:57:57.184780 28664 external_mini_cluster.cc:1428] Setting key c32cb469ad8810a05d3bd06e874c0320
Aug 08 15:57:57 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
I20260808 15:57:57.231302 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:57.182244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:55739 (local address 127.27.254.62:41001)
0808 15:57:57.195915 (+ 13671us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:57.195922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:57.208525 (+ 12603us) server_negotiation.cc:407] Connection header received
0808 15:57:57.211696 (+ 3171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:57.211703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:57.211797 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:57.211913 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:57.214196 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:57.215397 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:57.217971 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:57.218665 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:57.218783 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:57.223708 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:57.223716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:57.223721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:57.223783 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:57.227268 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:57.228181 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:57.228187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:57.228190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:57.228336 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:57.228929 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:57.228934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:57.228938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:57.229289 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:57.229536 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:57.230669 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0808 15:57:57.230893 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13510,"thread_start_us":13684,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
I20260808 15:57:57.232954 8782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:57.181663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41001 (local address 127.27.254.1:55739)
0808 15:57:57.207945 (+ 26282us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:57.208033 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:57.211027 (+ 2994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:57.212388 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:57.212402 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:57.213078 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:57:57.213982 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:57.214010 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:57.215613 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:57.215619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:57.217781 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:57.217815 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:57.219133 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:57.219143 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:57.219457 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:57.220386 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:57:57.220420 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:57:57.223387 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:57:57.227445 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:57.227454 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:57.227472 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:57.228048 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:57.228505 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:57.228509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:57.228512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:57.228777 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:57.229640 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:57:57.229650 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:57:57.230104 (+ 454us) client_negotiation.cc:769] Sending connection context
0808 15:57:57.230526 (+ 422us) client_negotiation.cc:240] Negotiation successful
0808 15:57:57.231073 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25898,"thread_start_us":25654,"threads_started":1}
I20260808 15:57:57.235021 8780 heartbeater.cc:347] Connected to a master server at 127.27.254.62:41001
I20260808 15:57:57.235337 8780 heartbeater.cc:464] Registering TS with master...
I20260808 15:57:57.236227 8780 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:57:57.238909 8567 ts_manager.cc:194] Registered new tserver with Master: d01a69babd4f411989b3ee4cdf1ad438 (127.27.254.1:38325)
I20260808 15:57:57.240991 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:55739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:57:57.270689 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:41001
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:57:57 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
Aug 08 15:57:57 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
W20260808 15:57:57.706456 8786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:57.706946 8786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:57.707150 8786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:57.718585 8786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:57.718791 8786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:57.719033 8786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:57:57.734912 8786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.254.20:40397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:57.736971 8786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:57.738745 8786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:57.794987 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:57.796626 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:57.804178 8797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:57.804605 8786 server_base.cc:1034] running on GCE node
I20260808 15:57:57.805701 8786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:57.812065 8786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:57.813452 8786 hybrid_clock.cc:648] HybridClock initialized: now 1786204677813322 us; error 106 us; skew 500 ppm
I20260808 15:57:57.813920 8786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:57 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:57.821746 8786 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:57:57.835991 8786 webserver.cc:460] Webserver started at http://127.27.254.2:33067/ using document root <none> and password file <none>
I20260808 15:57:57.836825 8786 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:57.837008 8786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:57.837448 8786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:57 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:57.863355 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "e0e772bde5df4143960027c64a408a86"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "87db9e7305855b70465ef4997893586c"
server_key_iv: "463a81312bdf1726201fe925a6f0b43e"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:57.864641 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "e0e772bde5df4143960027c64a408a86"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "87db9e7305855b70465ef4997893586c"
server_key_iv: "463a81312bdf1726201fe925a6f0b43e"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:57.870541 8786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:57:57.891249 8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:57.893239 8786 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20260808 15:57:57.893509 8786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "e0e772bde5df4143960027c64a408a86"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "87db9e7305855b70465ef4997893586c"
server_key_iv: "463a81312bdf1726201fe925a6f0b43e"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:57.893821 8786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:57.917333 8786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:57.921479 8786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:57.921774 8786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:57.922744 8786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:57:57.924101 8786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:57:57.924173 8786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:57.924328 8786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:57:57.924383 8786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:57.992360 8786 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:43207
I20260808 15:57:57.992451 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:43207 every 8 connection(s)
I20260808 15:57:57.993732 8786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:57:57.999873 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 8786
I20260808 15:57:58.000216 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:57:58.001250 28664 external_mini_cluster.cc:1428] Setting key adf1b4592faf715a6c74deb352b97246
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204677, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:57:58.019208 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:57.996633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:55607 (local address 127.27.254.62:41001)
0808 15:57:57.997119 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:57.997125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:57.998190 (+ 1065us) server_negotiation.cc:407] Connection header received
0808 15:57:58.000374 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:58.000382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:58.000497 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:58.000618 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:58.002591 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.003774 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.006351 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.006963 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.007097 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.011618 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:58.011626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:58.011634 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:58.011679 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:58.015241 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.016159 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.016168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.016174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.016336 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.016913 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.016921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.016927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.017327 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:58.017522 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:58.018406 (+ 884us) server_negotiation.cc:299] Negotiation successful
0808 15:57:58.018694 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260808 15:57:58.020228 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:57.997206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41001 (local address 127.27.254.2:55607)
0808 15:57:57.998014 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:57.998060 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:57.999951 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:58.001082 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:58.001093 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:58.001690 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:57:58.002408 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:58.002435 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.003956 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.003964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:58.006194 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:58.006211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.007203 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.007213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:58.007553 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.008366 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:57:58.008398 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:57:58.011385 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:57:58.015403 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:58.015412 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:58.015430 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:58.016015 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:58.016461 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:58.016466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:58.016470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:58.016753 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:58.017502 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:57:58.017511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:57:58.017945 (+ 434us) client_negotiation.cc:769] Sending connection context
0808 15:57:58.018258 (+ 313us) client_negotiation.cc:240] Negotiation successful
0808 15:57:58.018845 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":364,"threads_started":1}
I20260808 15:57:58.021840 8923 heartbeater.cc:347] Connected to a master server at 127.27.254.62:41001
I20260808 15:57:58.022176 8923 heartbeater.cc:464] Registering TS with master...
I20260808 15:57:58.023036 8923 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:57:58.024704 8567 ts_manager.cc:194] Registered new tserver with Master: e0e772bde5df4143960027c64a408a86 (127.27.254.2:43207)
I20260808 15:57:58.025821 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:55607
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:57:58.058528 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:41001
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:57:58.244321 8780 heartbeater.cc:503] Master 127.27.254.62:41001 was elected leader, sending a full tablet report...
W20260808 15:57:58.383384 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:58.383702 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:58.383873 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:58.392194 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:58.392333 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:58.392520 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:57:58.403014 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.254.20:40397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:58.404371 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:58.405740 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:58.420353 8934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:58.420548 8937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:58.420423 8935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:58.421340 8929 server_base.cc:1034] running on GCE node
I20260808 15:57:58.421965 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:58.422981 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:58.424199 8929 hybrid_clock.cc:648] HybridClock initialized: now 1786204678424156 us; error 38 us; skew 500 ppm
I20260808 15:57:58.424516 8929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204678, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:58.427881 8929 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:57:58.429653 8929 webserver.cc:460] Webserver started at http://127.27.254.3:36933/ using document root <none> and password file <none>
I20260808 15:57:58.430544 8929 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:58.430684 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:57:58.431037 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204678, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:58.447968 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "ec512eb1bd01484eb0e52f1898aa7bd4"
format_stamp: "Formatted at 2026-08-08 15:57:58 on dist-test-slave-srgv"
server_key: "2b469df4a67778ef724b6378a8773392"
server_key_iv: "9b0a4288e182b4d2c4f42f8a27726a2e"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:58.449019 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "ec512eb1bd01484eb0e52f1898aa7bd4"
format_stamp: "Formatted at 2026-08-08 15:57:58 on dist-test-slave-srgv"
server_key: "2b469df4a67778ef724b6378a8773392"
server_key_iv: "9b0a4288e182b4d2c4f42f8a27726a2e"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:58.455648 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260808 15:57:58.470825 8944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:58.472209 8929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:57:58.472374 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "ec512eb1bd01484eb0e52f1898aa7bd4"
format_stamp: "Formatted at 2026-08-08 15:57:58 on dist-test-slave-srgv"
server_key: "2b469df4a67778ef724b6378a8773392"
server_key_iv: "9b0a4288e182b4d2c4f42f8a27726a2e"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:58.472594 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:58.496790 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:58.500389 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:58.500669 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:58.501605 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:57:58.502877 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:57:58.502947 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:58.503105 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:57:58.503155 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:58.535312 8929 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37769
I20260808 15:57:58.535351 9056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37769 every 8 connection(s)
I20260808 15:57:58.536603 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:57:58.540745 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 8929
I20260808 15:57:58.540913 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:57:58.541337 28664 external_mini_cluster.cc:1428] Setting key 016cb7de8c5d52c558614952825d19b8
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204678, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:57:58.555254 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:58.539044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:46603 (local address 127.27.254.62:41001)
0808 15:57:58.539509 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:58.539514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:58.540442 (+ 928us) server_negotiation.cc:407] Connection header received
0808 15:57:58.542527 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:58.542532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:58.542628 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:58.542738 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:58.544156 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.544864 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.546414 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.546821 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.546950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.550036 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:58.550042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:58.550051 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:58.550081 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:58.552435 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.553107 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.553112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.553115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.553234 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.553648 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.553652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.553655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.553902 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:58.554041 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:58.554649 (+ 608us) server_negotiation.cc:299] Negotiation successful
0808 15:57:58.554871 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":229,"threads_started":1}
I20260808 15:57:58.556201 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:58.539624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41001 (local address 127.27.254.3:46603)
0808 15:57:58.540283 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 15:57:58.540330 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:57:58.542205 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:57:58.542973 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:57:58.542981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:57:58.543443 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:57:58.543996 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:58.544011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.545005 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.545011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:58.546264 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:57:58.546275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.547031 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.547037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:57:58.547281 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.547908 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:57:58.547933 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:57:58.549849 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:57:58.552562 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:58.552570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:58.552583 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:58.552967 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:58.553336 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:57:58.553340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:57:58.553343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:57:58.553544 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:57:58.554016 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:57:58.554023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:57:58.554334 (+ 311us) client_negotiation.cc:769] Sending connection context
0808 15:57:58.554567 (+ 233us) client_negotiation.cc:240] Negotiation successful
0808 15:57:58.554997 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":250,"threads_started":1}
I20260808 15:57:58.557314 9057 heartbeater.cc:347] Connected to a master server at 127.27.254.62:41001
I20260808 15:57:58.557610 9057 heartbeater.cc:464] Registering TS with master...
I20260808 15:57:58.558279 9057 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:57:58.559595 8567 ts_manager.cc:194] Registered new tserver with Master: ec512eb1bd01484eb0e52f1898aa7bd4 (127.27.254.3:37769)
I20260808 15:57:58.560354 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:46603
I20260808 15:57:58.567896 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:57:58.569542 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 8647
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:57:58.585951 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:58.570518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.27.254.1:38325)
0808 15:57:58.571147 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:58.571153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:58.571174 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:57:58.571224 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:58.571229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:58.571390 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:58.571511 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:58.572920 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.574008 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.575498 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.576185 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.576287 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.580719 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:58.580730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:58.580743 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:58.580779 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:58.583433 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.584054 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.584060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.584062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.584188 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.584474 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.584477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.584479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.584753 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:58.584991 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:58.585342 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 15:57:58.585582 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
I20260808 15:57:58.587106 8775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:58.667626 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 8647
I20260808 15:57:58.683575 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 8786
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:57:58.699427 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:58.684383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.27.254.2:43207)
0808 15:57:58.684932 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:58.684939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:58.684964 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:57:58.685023 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:58.685029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:58.685246 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:58.685381 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:58.686693 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.687667 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.688907 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.689485 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.689678 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.693983 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:58.693994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:58.694008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:58.694037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:58.696611 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.697247 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.697253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.697256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.697380 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.697978 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.697984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.697988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.698310 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:58.698517 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:58.698831 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 15:57:58.699064 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
I20260808 15:57:58.700624 8918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:58.781247 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 8786
I20260808 15:57:58.796775 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 8929
Aug 08 15:57:58 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:57:58.812577 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:57:58.797796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.27.254.3:37769)
0808 15:57:58.798465 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0808 15:57:58.798472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:57:58.798501 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:57:58.798568 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:57:58.798574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:57:58.798800 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:57:58.798929 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:57:58.800156 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.801145 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.802465 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:57:58.803006 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:57:58.803110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:57:58.807311 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:57:58.807322 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:57:58.807335 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:57:58.807366 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:57:58.810018 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.810674 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.810681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.810683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.810801 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:57:58.811145 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:57:58.811148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:57:58.811150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:57:58.811504 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:57:58.811744 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 15:57:58.812009 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 15:57:58.812186 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":357,"threads_started":1}
I20260808 15:57:58.813841 9052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:58.895423 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 8929
I20260808 15:57:58.911273 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 8535
I20260808 15:57:58.912459 8597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:57:59.017204 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 8535
I20260808 15:57:59.040362 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41001
--webserver_interface=127.27.254.62
--webserver_port=39365
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:41001
--ranger_config_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:57:59.364573 9069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:59.364928 9069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:59.365093 9069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:59.372689 9069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:57:59.372843 9069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:59.372947 9069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:59.373051 9069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:57:59.373155 9069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:57:59.384069 9069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:41001
--ranger_config_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:59.385445 9069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:59.387102 9069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:59.399358 9074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:59.399523 9077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:59.399549 9075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:59.400485 9069 server_base.cc:1034] running on GCE node
I20260808 15:57:59.401289 9069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:59.402683 9069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:59.403947 9069 hybrid_clock.cc:648] HybridClock initialized: now 1786204679403899 us; error 51 us; skew 500 ppm
I20260808 15:57:59.404315 9069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:59 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:57:59.407588 9069 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:57:59.409039 9069 webserver.cc:460] Webserver started at http://127.27.254.62:39365/ using document root <none> and password file <none>
I20260808 15:57:59.409750 9069 fs_manager.cc:359] Metadata directory not provided
I20260808 15:57:59.409927 9069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:57:59 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:57:59.458431 9069 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 15:57:59.461390 9085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:57:59.462925 9069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:57:59.463107 9069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "5a47501ba7cf4764b7d084999b39f8b8"
format_stamp: "Formatted at 2026-08-08 15:57:55 on dist-test-slave-srgv"
server_key: "5e910220ce3bdb43ff5083d8ba710f81"
server_key_iv: "a6024411aa7d056bfcce070db14d843a"
server_key_version: "kuduclusterkey@0"
I20260808 15:57:59.464020 9069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:57:59.485242 9069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:57:59.489038 9069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:57:59.489336 9069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:57:59.507922 9069 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:41001
I20260808 15:57:59.507943 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:41001 every 8 connection(s)
I20260808 15:57:59.509608 9069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:57:59.514302 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9069
I20260808 15:57:59.515120 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:38325
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:41001
--webserver_port=40207
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:57:59.516901 9137 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:57:59.520200 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Bootstrap starting.
I20260808 15:57:59.538502 9137 log.cc:824] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:57:59.547330 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:57:59.547878 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Bootstrap complete.
I20260808 15:57:59.556716 9137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:59.557642 9137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a47501ba7cf4764b7d084999b39f8b8, State: Initialized, Role: FOLLOWER
I20260808 15:57:59.558205 9137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:59.558456 9137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:57:59.558583 9137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:57:59.558764 9137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:57:59.562510 9137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:59.562986 9137 leader_election.cc:302] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a47501ba7cf4764b7d084999b39f8b8; no voters:
I20260808 15:57:59.563637 9137 leader_election.cc:288] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:57:59.563776 9141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:57:59.565110 9141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [term 2 LEADER]: Becoming Leader. State: Replica: 5a47501ba7cf4764b7d084999b39f8b8, State: Running, Role: LEADER
I20260808 15:57:59.565711 9141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } }
I20260808 15:57:59.565913 9137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:57:59.568215 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a47501ba7cf4764b7d084999b39f8b8. Latest consensus state: current_term: 2 leader_uuid: "5a47501ba7cf4764b7d084999b39f8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } } }
I20260808 15:57:59.568567 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:57:59.568538 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5a47501ba7cf4764b7d084999b39f8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a47501ba7cf4764b7d084999b39f8b8" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41001 } } }
I20260808 15:57:59.568964 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:57:59.579972 9149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:57:59.587052 9149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:57:59.588455 9149 catalog_manager.cc:1259] Loaded cluster ID: 368d7f858cf445a6b2231a5c7fa5e67e
I20260808 15:57:59.588627 9149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:57:59.593820 9149 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:57:59.595402 9149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Loaded TSK: 0
I20260808 15:57:59.596958 9149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:57:59.600234 9137 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260808 15:57:59.912209 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:57:59.912514 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:57:59.912714 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:57:59.924331 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:57:59.924582 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:57:59.924835 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:57:59.939800 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:38325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=40207
--tserver_master_addrs=127.27.254.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:57:59.941753 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:57:59.951166 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:57:59.983199 9175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:59.983870 9174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:57:59.984151 9177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:57:59.985673 9139 server_base.cc:1034] running on GCE node
I20260808 15:57:59.986598 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:57:59.995285 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:57:59.998726 9139 hybrid_clock.cc:648] HybridClock initialized: now 1786204679998649 us; error 67 us; skew 500 ppm
I20260808 15:57:59.999226 9139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:57:59 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:00.003646 9139 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:58:00.023432 9139 webserver.cc:460] Webserver started at http://127.27.254.1:40207/ using document root <none> and password file <none>
I20260808 15:58:00.024649 9139 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:00.024921 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:58:00 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:58:00.057154 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 15:58:00.075169 9185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:00.077297 9139 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.011s
I20260808 15:58:00.077630 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "d01a69babd4f411989b3ee4cdf1ad438"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "e9069e4387a23a8a7711fa44ad66290a"
server_key_iv: "6ed184e4c4d9e63d6d326fe03fcc0048"
server_key_version: "kuduclusterkey@0"
I20260808 15:58:00.079258 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:00.139816 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:00.145594 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:00.145996 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:00.153489 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:00.161737 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:00.161948 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:00.162128 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:00.162246 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:00.250296 9139 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:38325
I20260808 15:58:00.250387 9298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:38325 every 8 connection(s)
I20260808 15:58:00.252256 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:58:00.260620 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9139
I20260808 15:58:00.261566 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:43207
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:41001
--webserver_port=33067
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:58:00 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:00.288383 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:00.255948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41001 (local address 127.27.254.1:50957)
0808 15:58:00.261431 (+ 5483us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:00.261502 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:00.264403 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:00.266562 (+ 2159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:00.266574 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:00.267393 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:00.268244 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:00.268272 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:00.270761 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:00.270766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:00.272344 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:00.272361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:00.274146 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:00.274150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:00.274324 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:00.275026 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:00.275048 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:00.277576 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:00.282369 (+ 4793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:00.282380 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:00.282398 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:00.283094 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:00.283752 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:00.283759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:00.283763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:00.284023 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:00.285244 (+ 1221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:00.285251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:00.285849 (+ 598us) client_negotiation.cc:769] Sending connection context
0808 15:58:00.286229 (+ 380us) client_negotiation.cc:240] Negotiation successful
0808 15:58:00.286837 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5123,"thread_start_us":4903,"threads_started":1}
I20260808 15:58:00.291119 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:00.255861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:50957 (local address 127.27.254.62:41001)
0808 15:58:00.256701 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:00.256726 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:00.261676 (+ 4950us) server_negotiation.cc:407] Connection header received
0808 15:58:00.265221 (+ 3545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:00.265247 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:00.265817 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:00.266278 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:00.268535 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:00.270584 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:00.272529 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:00.273658 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:00.273777 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:00.277890 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:00.277903 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:00.277924 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:00.277972 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:00.282170 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:00.283347 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:00.283357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:00.283365 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:00.283595 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:00.284174 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:00.284179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:00.284182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:00.285074 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:00.288960 (+ 3886us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:00.289279 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 15:58:00.289767 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":319,"threads_started":1}
I20260808 15:58:00.295250 9299 heartbeater.cc:347] Connected to a master server at 127.27.254.62:41001
I20260808 15:58:00.295619 9299 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:00.296463 9299 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
W20260808 15:58:00.299588 9299 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:41001 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:58:00.299798 9299 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:00.300105 9299 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:58:00.301348 9299 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:00.301613 9299 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
W20260808 15:58:00.302598 9299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:58:00.706462 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:00.706773 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:00.706987 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:00.717285 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:00.717447 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:00.717633 9303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:58:00.731932 9303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:43207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=33067
--tserver_master_addrs=127.27.254.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:00.733599 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:00.735220 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:00.751858 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:00.752116 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:00.753875 9303 server_base.cc:1034] running on GCE node
W20260808 15:58:00.754487 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:00.755280 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:00.756532 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:00.757830 9303 hybrid_clock.cc:648] HybridClock initialized: now 1786204680757759 us; error 51 us; skew 500 ppm
I20260808 15:58:00.758215 9303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:00 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:00.762473 9303 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:58:00.764663 9303 webserver.cc:460] Webserver started at http://127.27.254.2:33067/ using document root <none> and password file <none>
I20260808 15:58:00.765996 9303 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:00.766312 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:58:00 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:58:00.806304 9303 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260808 15:58:00.810745 9330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:00.812577 9303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:58:00.812800 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "e0e772bde5df4143960027c64a408a86"
format_stamp: "Formatted at 2026-08-08 15:57:57 on dist-test-slave-srgv"
server_key: "87db9e7305855b70465ef4997893586c"
server_key_iv: "463a81312bdf1726201fe925a6f0b43e"
server_key_version: "kuduclusterkey@0"
I20260808 15:58:00.814281 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:00.852665 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:00.857975 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:00.858278 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:00.859556 9303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:00.861231 9303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:00.861332 9303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:00.861451 9303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:00.861533 9303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 15:58:00 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:00.973726 9303 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:43207
I20260808 15:58:00.973891 9443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:43207 every 8 connection(s)
I20260808 15:58:00.975556 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:58:00.978466 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9303
I20260808 15:58:00.979234 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.254.20:40397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:37769
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:41001
--webserver_port=36933
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:58:00 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:01.012036 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:00.979831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:44393 (local address 127.27.254.62:41001)
0808 15:58:00.980903 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:00.980909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:00.989291 (+ 8382us) server_negotiation.cc:407] Connection header received
0808 15:58:00.989387 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:00.989393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:00.989508 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:00.989632 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:00.995606 (+ 5974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:00.996803 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.000228 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.000647 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.000728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:01.005003 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:01.005013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:01.005020 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:01.005068 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:01.008911 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.009698 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.009704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.009707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.009848 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.010270 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.010275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.010278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.010550 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:01.010703 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:01.011446 (+ 743us) server_negotiation.cc:299] Negotiation successful
0808 15:58:01.011658 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":829,"threads_started":1}
I20260808 15:58:01.013192 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:00.979048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41001 (local address 127.27.254.2:44393)
0808 15:58:00.985537 (+ 6489us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:00.985594 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:00.987989 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:00.993823 (+ 5834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:00.993837 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:00.994487 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:00.995366 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:00.995395 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:00.997898 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:00.997904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:01.000013 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:01.000039 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.001040 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.001044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:01.001234 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:01.002060 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:01.002099 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:01.004739 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:01.009054 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:01.009064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:01.009080 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:01.009573 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:01.009963 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:01.009968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:01.009972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:01.010151 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:01.010677 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:01.010685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:01.011076 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 15:58:01.011341 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 15:58:01.011888 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4573,"thread_start_us":4359,"threads_started":1}
I20260808 15:58:01.014808 9444 heartbeater.cc:347] Connected to a master server at 127.27.254.62:41001
I20260808 15:58:01.015116 9444 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:01.015833 9444 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:58:01.017656 9102 ts_manager.cc:194] Registered new tserver with Master: e0e772bde5df4143960027c64a408a86 (127.27.254.2:43207)
I20260808 15:58:01.019794 9102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:44393
I20260808 15:58:01.302774 9299 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:01.303236 9299 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:58:01.304649 9102 ts_manager.cc:194] Registered new tserver with Master: d01a69babd4f411989b3ee4cdf1ad438 (127.27.254.1:38325)
I20260808 15:58:01.305986 9102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:50957
Aug 08 15:58:01 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
W20260808 15:58:01.436408 9448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:01.436785 9448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:01.436941 9448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:01.448225 9448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:01.448371 9448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:01.448550 9448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:58:01.463747 9448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.254.20:40397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:37769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=36933
--tserver_master_addrs=127.27.254.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:01.465359 9448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:01.467080 9448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:01.484063 9454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:01.484165 9453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:01.491286 9456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:01.492058 9448 server_base.cc:1034] running on GCE node
I20260808 15:58:01.492794 9448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:01.494041 9448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:01.495265 9448 hybrid_clock.cc:648] HybridClock initialized: now 1786204681495205 us; error 50 us; skew 500 ppm
I20260808 15:58:01.495621 9448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:01 dist-test-slave-srgv krb5kdc[8091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:01.499538 9448 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:58:01.506916 9448 webserver.cc:460] Webserver started at http://127.27.254.3:36933/ using document root <none> and password file <none>
I20260808 15:58:01.510196 9448 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:01.510373 9448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:58:01 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:58:01.549865 9448 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.004s
I20260808 15:58:01.553743 9464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:01.555586 9448 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:58:01.555819 9448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "ec512eb1bd01484eb0e52f1898aa7bd4"
format_stamp: "Formatted at 2026-08-08 15:57:58 on dist-test-slave-srgv"
server_key: "2b469df4a67778ef724b6378a8773392"
server_key_iv: "9b0a4288e182b4d2c4f42f8a27726a2e"
server_key_version: "kuduclusterkey@0"
I20260808 15:58:01.557209 9448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:01.585256 9448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:01.590411 9448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:01.590767 9448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:01.592643 9448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:01.601027 9448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:01.601202 9448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:01.601344 9448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:01.601480 9448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:58:01 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for HTTP/127.27.254.20@KRBTEST.COM
I20260808 15:58:01.897311 9448 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37769
I20260808 15:58:01.897351 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37769 every 8 connection(s)
I20260808 15:58:01.899129 9448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:58:01.903393 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9448
Aug 08 15:58:01 dist-test-slave-srgv krb5kdc[8091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:01.923213 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.901843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:45019 (local address 127.27.254.62:41001)
0808 15:58:01.902348 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:01.902355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:01.905453 (+ 3098us) server_negotiation.cc:407] Connection header received
0808 15:58:01.906686 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:01.906691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:01.906773 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:01.906925 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:01.908358 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.909075 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.911361 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.912076 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.912181 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:01.915996 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:01.916000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:01.916003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:01.916065 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:01.919778 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.920617 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.920623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.920628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.920772 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.921200 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.921204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.921208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.921573 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:01.921754 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:01.922553 (+ 799us) server_negotiation.cc:299] Negotiation successful
0808 15:58:01.922794 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
I20260808 15:58:01.924425 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.902425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41001 (local address 127.27.254.3:45019)
0808 15:58:01.904528 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:01.904577 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:01.906416 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:01.907158 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:01.907166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:01.907637 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:01.908193 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:01.908209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.909221 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.909226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:01.911161 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:01.911190 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.912499 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.912503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:01.912682 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:01.913329 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:01.913349 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:01.915777 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:01.919929 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:01.919938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:01.919955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:01.920448 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:01.920896 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:01.920900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:01.920904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:01.921075 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:01.921847 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:01.921858 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:01.922187 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 15:58:01.922454 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 15:58:01.923005 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1832,"thread_start_us":1648,"threads_started":1}
I20260808 15:58:01.926211 9586 heartbeater.cc:347] Connected to a master server at 127.27.254.62:41001
I20260808 15:58:01.926543 9586 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:01.927059 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.904875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.27.254.62:41001)
0808 15:58:01.905364 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:01.905371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:01.910053 (+ 4682us) server_negotiation.cc:407] Connection header received
0808 15:58:01.910231 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:01.910236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:01.910313 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:01.910432 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:01.912432 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.913955 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.915549 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.916247 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.916387 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:01.919775 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:01.919782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:01.919787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:01.919829 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:01.923529 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.924320 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.924326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.924329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.924489 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.925315 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.925320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.925324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.925695 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:01.925989 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:01.926307 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 15:58:01.926533 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260808 15:58:01.927295 9586 heartbeater.cc:511] Master 127.27.254.62:41001 requested a full tablet report, sending...
I20260808 15:58:01.928808 9101 ts_manager.cc:194] Registered new tserver with Master: ec512eb1bd01484eb0e52f1898aa7bd4 (127.27.254.3:37769)
I20260808 15:58:01.930166 9101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:45019
I20260808 15:58:01.941886 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:58:01.963837 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.945631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.27.254.62:41001)
0808 15:58:01.945999 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:01.946009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:01.948649 (+ 2640us) server_negotiation.cc:407] Connection header received
0808 15:58:01.948860 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:01.948866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:01.948956 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:01.949026 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:01.951222 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.952329 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.954329 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:01.954755 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:01.954892 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:01.958188 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:01.958193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:01.958196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:01.958230 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:01.960844 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.961646 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.961650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.961652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.961778 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:01.962293 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:01.962300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:01.962308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:01.962614 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:01.962895 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:01.963245 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 15:58:01.963472 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 15:58:01.969271 9101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:58:01.979130 9101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:58:02.008152 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.994030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:43207 (local address 127.0.0.1:54492)
0808 15:58:01.995771 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:01.995794 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:01.995958 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:01.997210 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:01.997222 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:01.997262 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:01.997852 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:01.997879 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.000397 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.000406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.003638 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.003660 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.007053 (+ 3393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.007060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.007249 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.007300 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 15:58:02.007447 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 15:58:02.007667 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":1849,"threads_started":1}
I20260808 15:58:02.009142 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.994650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.27.254.2:43207)
0808 15:58:01.995277 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:01.995290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:01.996106 (+ 816us) server_negotiation.cc:407] Connection header received
0808 15:58:01.996243 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:01.996254 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:01.996563 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:01.996793 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:01.998193 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.000171 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.003893 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.006003 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.006165 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.006480 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.008618 (+ 2138us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.008802 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":322,"threads_started":1}
I20260808 15:58:02.013223 9378 tablet_service.cc:1467] Processing CreateTablet for tablet 8564e71db16c4ec1904f9a853c746762 (DEFAULT_TABLE table=test-table [id=3ab527b7a39241df849bcf1b82012d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:02.014719 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.993536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:38325 (local address 127.0.0.1:45528)
0808 15:58:01.994431 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:01.994487 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:01.994672 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:02.007629 (+ 12957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:02.007634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:02.007657 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:02.008074 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.008093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.010316 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.010324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.012149 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.012160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.013875 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.013881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.014057 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.014091 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:58:02.014189 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:58:02.014351 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":30,"thread_start_us":404,"threads_started":1}
I20260808 15:58:02.014925 9378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8564e71db16c4ec1904f9a853c746762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:02.014755 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.993881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.27.254.1:38325)
0808 15:58:02.006985 (+ 13104us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.006994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.007028 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:58:02.007134 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.007143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.007417 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.008342 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:02.008589 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.010105 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.012363 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.013652 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.013790 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.014021 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.014238 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.014398 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12919,"thread_start_us":12753,"threads_started":1}
I20260808 15:58:02.015605 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.994893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37769 (local address 127.0.0.1:42622)
0808 15:58:02.006079 (+ 11186us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:02.006104 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:02.006251 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:02.007905 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:02.007909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:02.007931 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:02.008303 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.008313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.010092 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.010101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.012767 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.012780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.014897 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.014903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.015065 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.015111 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 15:58:02.015206 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:58:02.015343 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11011,"mutex_wait_us":1,"thread_start_us":11440,"threads_started":1}
I20260808 15:58:02.016189 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:01.995615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.27.254.3:37769)
0808 15:58:02.006992 (+ 11377us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.007002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.007036 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:58:02.007137 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.007145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.007417 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.007618 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:02.008503 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.009967 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.012932 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.014396 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.014523 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.014735 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.015826 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.015947 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11189,"thread_start_us":11896,"threads_started":1}
I20260808 15:58:02.020022 9234 tablet_service.cc:1467] Processing CreateTablet for tablet 8564e71db16c4ec1904f9a853c746762 (DEFAULT_TABLE table=test-table [id=3ab527b7a39241df849bcf1b82012d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:02.020022 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 8564e71db16c4ec1904f9a853c746762 (DEFAULT_TABLE table=test-table [id=3ab527b7a39241df849bcf1b82012d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:02.021456 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8564e71db16c4ec1904f9a853c746762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:02.021485 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8564e71db16c4ec1904f9a853c746762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:02.024212 9444 heartbeater.cc:503] Master 127.27.254.62:41001 was elected leader, sending a full tablet report...
I20260808 15:58:02.029441 9607 tablet_bootstrap.cc:492] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Bootstrap starting.
I20260808 15:58:02.032541 9607 tablet_bootstrap.cc:654] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:02.033589 9607 log.cc:824] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:02.036552 9607 tablet_bootstrap.cc:492] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: No bootstrap required, opened a new log
I20260808 15:58:02.036783 9610 tablet_bootstrap.cc:492] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Bootstrap starting.
I20260808 15:58:02.037004 9607 ts_tablet_manager.cc:1397] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260808 15:58:02.040434 9610 tablet_bootstrap.cc:654] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:02.041282 9609 tablet_bootstrap.cc:492] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Bootstrap starting.
I20260808 15:58:02.042047 9610 log.cc:824] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:02.044972 9609 tablet_bootstrap.cc:654] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:02.045775 9610 tablet_bootstrap.cc:492] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: No bootstrap required, opened a new log
I20260808 15:58:02.046041 9610 ts_tablet_manager.cc:1397] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:58:02.046295 9609 log.cc:824] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:02.047433 9607 raft_consensus.cc:348] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.047878 9607 raft_consensus.cc:374] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:02.047973 9607 raft_consensus.cc:729] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e772bde5df4143960027c64a408a86, State: Initialized, Role: FOLLOWER
I20260808 15:58:02.048609 9607 consensus_queue.cc:260] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.049042 9609 tablet_bootstrap.cc:492] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: No bootstrap required, opened a new log
I20260808 15:58:02.049345 9609 ts_tablet_manager.cc:1397] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:58:02.050504 9607 ts_tablet_manager.cc:1428] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 15:58:02.056631 9610 raft_consensus.cc:348] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.056984 9610 raft_consensus.cc:374] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:02.057075 9610 raft_consensus.cc:729] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec512eb1bd01484eb0e52f1898aa7bd4, State: Initialized, Role: FOLLOWER
I20260808 15:58:02.057682 9610 consensus_queue.cc:260] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.058123 9609 raft_consensus.cc:348] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.058719 9586 heartbeater.cc:503] Master 127.27.254.62:41001 was elected leader, sending a full tablet report...
I20260808 15:58:02.059327 9610 ts_tablet_manager.cc:1428] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260808 15:58:02.059602 9609 raft_consensus.cc:374] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:02.059724 9609 raft_consensus.cc:729] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01a69babd4f411989b3ee4cdf1ad438, State: Initialized, Role: FOLLOWER
I20260808 15:58:02.060322 9609 consensus_queue.cc:260] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.061347 9299 heartbeater.cc:503] Master 127.27.254.62:41001 was elected leader, sending a full tablet report...
I20260808 15:58:02.061872 9609 ts_tablet_manager.cc:1428] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
W20260808 15:58:02.150817 9587 tablet.cc:2369] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:58:02.234661 9445 tablet.cc:2369] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:58:02.264865 9300 tablet.cc:2369] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:02.376353 9615 raft_consensus.cc:482] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:02.376662 9615 raft_consensus.cc:504] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.378366 9615 leader_election.cc:288] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec512eb1bd01484eb0e52f1898aa7bd4 (127.27.254.3:37769), e0e772bde5df4143960027c64a408a86 (127.27.254.2:43207)
I20260808 15:58:02.387585 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.378784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:36427 (local address 127.27.254.3:37769)
0808 15:58:02.379099 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.379102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.379847 (+ 745us) server_negotiation.cc:407] Connection header received
0808 15:58:02.380041 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.380046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.380128 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.380232 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:02.381047 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.382129 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.385147 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.386403 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.386553 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.386626 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.387183 (+ 557us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.387326 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260808 15:58:02.387589 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.378784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:60819 (local address 127.27.254.2:43207)
0808 15:58:02.378977 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.378981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.379847 (+ 866us) server_negotiation.cc:407] Connection header received
0808 15:58:02.380041 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.380046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.380145 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.380236 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:02.381047 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.382126 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.385146 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.386411 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.386553 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.386626 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.387183 (+ 557us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.387343 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:58:02.387840 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.378869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37769 (local address 127.27.254.1:36427)
0808 15:58:02.379697 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:02.379717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:02.379892 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:02.380267 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:02.380272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:02.380313 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:02.380885 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.380898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.382287 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.382291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.384994 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.385009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.386697 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.386701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.386960 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.387030 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 15:58:02.387127 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 15:58:02.387346 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":279,"spinlock_wait_cycles":12544,"thread_start_us":459,"threads_started":1}
I20260808 15:58:02.388116 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.378865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:43207 (local address 127.27.254.1:60819)
0808 15:58:02.379733 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:02.379750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:02.379911 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:02.380267 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:02.380272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:02.380315 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:02.380888 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.380898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.382285 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.382290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.384994 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:02.385009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.386683 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.386689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:02.386964 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.387029 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 15:58:02.387127 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:58:02.387346 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":407,"spinlock_wait_cycles":48384,"thread_start_us":428,"threads_started":1}
I20260808 15:58:02.388877 9541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8564e71db16c4ec1904f9a853c746762" candidate_uuid: "d01a69babd4f411989b3ee4cdf1ad438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" is_pre_election: true
I20260808 15:58:02.388978 9398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8564e71db16c4ec1904f9a853c746762" candidate_uuid: "d01a69babd4f411989b3ee4cdf1ad438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e772bde5df4143960027c64a408a86" is_pre_election: true
I20260808 15:58:02.389376 9541 raft_consensus.cc:2454] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01a69babd4f411989b3ee4cdf1ad438 in term 0.
I20260808 15:58:02.389519 9398 raft_consensus.cc:2454] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01a69babd4f411989b3ee4cdf1ad438 in term 0.
I20260808 15:58:02.390079 9186 leader_election.cc:302] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01a69babd4f411989b3ee4cdf1ad438, ec512eb1bd01484eb0e52f1898aa7bd4; no voters:
I20260808 15:58:02.390486 9615 raft_consensus.cc:2781] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:58:02.390601 9615 raft_consensus.cc:482] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:58:02.390669 9615 raft_consensus.cc:3037] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:02.392794 9615 raft_consensus.cc:504] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.393337 9615 leader_election.cc:288] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [CANDIDATE]: Term 1 election: Requested vote from peers ec512eb1bd01484eb0e52f1898aa7bd4 (127.27.254.3:37769), e0e772bde5df4143960027c64a408a86 (127.27.254.2:43207)
I20260808 15:58:02.393707 9541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8564e71db16c4ec1904f9a853c746762" candidate_uuid: "d01a69babd4f411989b3ee4cdf1ad438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4"
I20260808 15:58:02.393803 9398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8564e71db16c4ec1904f9a853c746762" candidate_uuid: "d01a69babd4f411989b3ee4cdf1ad438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e772bde5df4143960027c64a408a86"
I20260808 15:58:02.393934 9541 raft_consensus.cc:3037] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:02.394006 9398 raft_consensus.cc:3037] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:02.395967 9541 raft_consensus.cc:2454] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01a69babd4f411989b3ee4cdf1ad438 in term 1.
I20260808 15:58:02.396008 9398 raft_consensus.cc:2454] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01a69babd4f411989b3ee4cdf1ad438 in term 1.
I20260808 15:58:02.396311 9186 leader_election.cc:302] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01a69babd4f411989b3ee4cdf1ad438, ec512eb1bd01484eb0e52f1898aa7bd4; no voters:
I20260808 15:58:02.396559 9615 raft_consensus.cc:2781] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:02.396982 9615 raft_consensus.cc:686] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 1 LEADER]: Becoming Leader. State: Replica: d01a69babd4f411989b3ee4cdf1ad438, State: Running, Role: LEADER
I20260808 15:58:02.397522 9615 consensus_queue.cc:237] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } }
I20260808 15:58:02.402189 9101 catalog_manager.cc:5495] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 reported cstate change: term changed from 0 to 1, leader changed from <none> to d01a69babd4f411989b3ee4cdf1ad438 (127.27.254.1). New cstate: current_term: 1 leader_uuid: "d01a69babd4f411989b3ee4cdf1ad438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01a69babd4f411989b3ee4cdf1ad438" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 38325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260808 15:58:02.461015 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.455672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.27.254.1:38325)
0808 15:58:02.455897 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.455903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.455922 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:58:02.456135 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.456139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.456227 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.456385 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:58:02.456771 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.457562 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.458926 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.459314 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.459455 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.459962 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:58:02.460450 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:58:02.460614 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.460669 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.460744 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:58:02.469022 9541 raft_consensus.cc:1261] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 1 FOLLOWER]: Refusing update from remote peer d01a69babd4f411989b3ee4cdf1ad438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:02.469043 9398 raft_consensus.cc:1261] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 1 FOLLOWER]: Refusing update from remote peer d01a69babd4f411989b3ee4cdf1ad438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:02.469978 9618 consensus_queue.cc:1035] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec512eb1bd01484eb0e52f1898aa7bd4" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:02.470345 9615 consensus_queue.cc:1035] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e772bde5df4143960027c64a408a86" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:02.480903 9621 mvcc.cc:205] Tried to move back new op lower bound from 7316294379383635968 to 7316294379104202752. Current Snapshot: MvccSnapshot[applied={T|T < 7316294379383635968}]
I20260808 15:58:02.487421 9623 mvcc.cc:205] Tried to move back new op lower bound from 7316294379383635968 to 7316294379104202752. Current Snapshot: MvccSnapshot[applied={T|T < 7316294379383635968}]
I20260808 15:58:02.489971 9622 mvcc.cc:205] Tried to move back new op lower bound from 7316294379383635968 to 7316294379104202752. Current Snapshot: MvccSnapshot[applied={T|T < 7316294379383635968}]
I20260808 15:58:02.501997 9102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:02.502274 9102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:02.505793 9102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a47501ba7cf4764b7d084999b39f8b8: Sending DeleteTablet for 3 replicas of tablet 8564e71db16c4ec1904f9a853c746762
I20260808 15:58:02.506840 9234 tablet_service.cc:1514] Processing DeleteTablet for tablet 8564e71db16c4ec1904f9a853c746762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:02 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:45528
I20260808 15:58:02.506932 9514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8564e71db16c4ec1904f9a853c746762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:02 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:42622
I20260808 15:58:02.507040 9378 tablet_service.cc:1514] Processing DeleteTablet for tablet 8564e71db16c4ec1904f9a853c746762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:02 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:54492
I20260808 15:58:02.511159 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9139
I20260808 15:58:02.520794 9633 tablet_replica.cc:330] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: stopping tablet replica
I20260808 15:58:02.521231 9633 raft_consensus.cc:2229] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:02.522095 9633 raft_consensus.cc:2258] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:02.524209 9633 ts_tablet_manager.cc:1905] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:02.531073 9633 ts_tablet_manager.cc:1918] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:02.531234 9633 log.cc:1196] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/8564e71db16c4ec1904f9a853c746762
I20260808 15:58:02.531766 9633 ts_tablet_manager.cc:1939] T 8564e71db16c4ec1904f9a853c746762 P e0e772bde5df4143960027c64a408a86: Deleting consensus metadata
I20260808 15:58:02.533586 9089 catalog_manager.cc:4841] TS e0e772bde5df4143960027c64a408a86 (127.27.254.2:43207): tablet 8564e71db16c4ec1904f9a853c746762 (table test-table [id=3ab527b7a39241df849bcf1b82012d09]) successfully deleted
I20260808 15:58:02.538409 9632 tablet_replica.cc:330] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: stopping tablet replica
I20260808 15:58:02.538992 9632 raft_consensus.cc:2229] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:02.539523 9632 raft_consensus.cc:2258] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:02.541795 9631 tablet_replica.cc:330] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: stopping tablet replica
I20260808 15:58:02.542030 9632 ts_tablet_manager.cc:1905] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:02.542191 9631 raft_consensus.cc:2229] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:58:02.542835 9631 raft_consensus.cc:2258] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:02.544582 9631 ts_tablet_manager.cc:1905] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:02.549707 9632 ts_tablet_manager.cc:1918] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:02.549926 9632 log.cc:1196] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/8564e71db16c4ec1904f9a853c746762
I20260808 15:58:02.550511 9632 ts_tablet_manager.cc:1939] T 8564e71db16c4ec1904f9a853c746762 P ec512eb1bd01484eb0e52f1898aa7bd4: Deleting consensus metadata
I20260808 15:58:02.552243 9086 catalog_manager.cc:4841] TS ec512eb1bd01484eb0e52f1898aa7bd4 (127.27.254.3:37769): tablet 8564e71db16c4ec1904f9a853c746762 (table test-table [id=3ab527b7a39241df849bcf1b82012d09]) successfully deleted
I20260808 15:58:02.557533 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.512940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.27.254.1:38325)
0808 15:58:02.513158 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.513166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.535089 (+ 21923us) server_negotiation.cc:407] Connection header received
0808 15:58:02.535239 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.535246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.535359 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.535471 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:02.537162 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.538451 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.540233 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.540928 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.541060 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.549115 (+ 8055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:02.549128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:02.549145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:02.549188 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:02.553610 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:02.554292 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:02.554301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:02.554305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:02.554477 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:02.554889 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:02.554896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:02.554900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:02.555323 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:02.555633 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.555924 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.556190 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:58:02.557508 9631 ts_tablet_manager.cc:1918] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:02.557760 9631 log.cc:1196] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/8564e71db16c4ec1904f9a853c746762
I20260808 15:58:02.558261 9294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:02.558357 9631 ts_tablet_manager.cc:1939] T 8564e71db16c4ec1904f9a853c746762 P d01a69babd4f411989b3ee4cdf1ad438: Deleting consensus metadata
I20260808 15:58:02.671562 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9139
I20260808 15:58:02.671772 9086 catalog_manager.cc:4841] TS d01a69babd4f411989b3ee4cdf1ad438 (127.27.254.1:38325): tablet 8564e71db16c4ec1904f9a853c746762 (table test-table [id=3ab527b7a39241df849bcf1b82012d09]) successfully deleted
I20260808 15:58:02.693089 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9303
I20260808 15:58:02.706336 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.693956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.27.254.2:43207)
0808 15:58:02.694134 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.694142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.694162 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:58:02.694387 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.694392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.694461 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.694577 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:02.695979 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.696645 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.698016 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.698436 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.698570 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.700928 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:02.700941 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:02.700955 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:02.700990 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:02.703551 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:02.704225 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:02.704232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:02.704235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:02.704371 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:02.704730 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:02.704736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:02.704739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:02.705095 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:02.705401 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.705765 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.705991 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:58:02.706938 9439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:02.794119 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9303
I20260808 15:58:02.810927 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9448
I20260808 15:58:02.828617 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:02.811687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.27.254.3:37769)
0808 15:58:02.811848 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:02.811852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:02.811870 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:58:02.812036 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:02.812039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:02.812113 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:02.812203 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:02.813545 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.814249 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.815518 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:02.815874 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:02.815969 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:02.818427 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:02.818441 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:02.818456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:02.818486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:02.826238 (+ 7752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:02.826934 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:02.826942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:02.826944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:02.827075 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:02.827390 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:02.827394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:02.827395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:02.827733 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:02.827952 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:02.828200 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 15:58:02.828363 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:58:02.829278 9581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:02.917618 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9448
I20260808 15:58:02.934391 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9069
I20260808 15:58:02.935601 9132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:03.046574 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9069
I20260808 15:58:03.069665 28664 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 15:58:03.405042 28664 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 15:58:03.405229 28664 mini_ranger.cc:66] Stopping Ranger...
I20260808 15:58:03.743067 28664 mini_ranger.cc:68] Stopped Ranger
2026-08-08 15:58:03.743 UTC [8121] LOG: received smart shutdown request
2026-08-08 15:58:03.746 UTC [8121] LOG: background worker "logical replication launcher" (PID 8130) exited with exit code 1
2026-08-08 15:58:03.747 UTC [8125] LOG: shutting down
2026-08-08 15:58:03.901 UTC [8121] LOG: database system is shut down
2026-08-08T15:58:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51975 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:58:04 dist-test-slave-srgv krb5kdc[9647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:58:04 dist-test-slave-srgv krb5kdc[9647](info): set up 2 sockets
Aug 08 15:58:04 dist-test-slave-srgv krb5kdc[9647](info): commencing operation
krb5kdc: starting...
W20260808 15:58:05.135612 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:58:05 dist-test-slave-srgv krb5kdc[9647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:58:05.372434 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41405
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:41405,127.27.254.61:39745,127.27.254.60:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:05.698998 9660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:05.699277 9660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:05.699435 9660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:05.706630 9660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:05.706736 9660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:05.706835 9660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:05.706943 9660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:05.707027 9660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:05.716853 9660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:41405,127.27.254.61:39745,127.27.254.60:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:05.718202 9660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:05.719532 9660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:05.732487 9665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:05.732652 9666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:05.732618 9668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:05.733983 9660 server_base.cc:1034] running on GCE node
I20260808 15:58:05.734781 9660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:05.736158 9660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:05.737380 9660 hybrid_clock.cc:648] HybridClock initialized: now 1786204685737324 us; error 56 us; skew 500 ppm
I20260808 15:58:05.737766 9660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:05 dist-test-slave-srgv krb5kdc[9647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:05.740852 9660 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:05.742255 9660 webserver.cc:460] Webserver started at http://127.27.254.62:34669/ using document root <none> and password file <none>
I20260808 15:58:05.742879 9660 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:05.742977 9660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:05.743212 9660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:05.746085 9660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "e509965a59a345f29d4fd2ec6387eb3f"
format_stamp: "Formatted at 2026-08-08 15:58:05 on dist-test-slave-srgv"
server_key: "0ecbb12b65008de980985c1fbea9568e"
server_key_iv: "70387ccc313b94a05ced79e59f797487"
server_key_version: "encryptionkey@0"
I20260808 15:58:05.746841 9660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "e509965a59a345f29d4fd2ec6387eb3f"
format_stamp: "Formatted at 2026-08-08 15:58:05 on dist-test-slave-srgv"
server_key: "0ecbb12b65008de980985c1fbea9568e"
server_key_iv: "70387ccc313b94a05ced79e59f797487"
server_key_version: "encryptionkey@0"
I20260808 15:58:05.752267 9660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:58:05.755599 9675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:05.756985 9660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:58:05.757194 9660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "e509965a59a345f29d4fd2ec6387eb3f"
format_stamp: "Formatted at 2026-08-08 15:58:05 on dist-test-slave-srgv"
server_key: "0ecbb12b65008de980985c1fbea9568e"
server_key_iv: "70387ccc313b94a05ced79e59f797487"
server_key_version: "encryptionkey@0"
I20260808 15:58:05.757391 9660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:05.769567 9660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:05.773857 9660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:05.774153 9660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:05.791563 9660 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:41405
I20260808 15:58:05.791584 9726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:41405 every 8 connection(s)
I20260808 15:58:05.792989 9660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:05.796335 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9660
I20260808 15:58:05.796566 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:58:05.796939 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:05.797278 28664 external_mini_cluster.cc:1428] Setting key 24e19b014f2aa7c3aab2763594837ca4
I20260808 15:58:05.803429 9727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:58:05 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.61@KRBTEST.COM" created.
I20260808 15:58:05.827669 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.806990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.27.254.62:41405)
0808 15:58:05.808740 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:05.808754 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:05.808807 (+ 53us) server_negotiation.cc:407] Connection header received
0808 15:58:05.810457 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:05.810467 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:05.810778 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:05.811000 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:05.812123 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:05.814199 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:05.815563 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:05.816598 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:05.816787 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:05.820181 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:05.820196 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:05.820215 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:05.820253 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:05.823394 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:05.824270 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:05.824276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:05.824278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:05.824413 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:05.824876 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:05.824879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:05.824881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:05.825355 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:05.825596 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:05.826257 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 15:58:05.826569 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":1531,"spinlock_wait_cycles":13184,"thread_start_us":1317,"threads_started":1}
I20260808 15:58:05.827696 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.807013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41405 (local address 127.0.0.1:51508)
0808 15:58:05.808009 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.808077 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:05.810087 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:05.811036 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:05.811041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:05.811596 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:05.811928 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:05.811943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:05.814364 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:05.814368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:05.815422 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:05.815433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:05.816834 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:05.816839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:05.816962 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:05.817776 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:05.817813 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:05.819898 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:05.823521 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:05.823535 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:05.823550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:05.824153 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:05.824523 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:05.824530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:05.824536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:05.824744 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:05.825577 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:05.825588 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:05.825862 (+ 274us) client_negotiation.cc:769] Sending connection context
0808 15:58:05.826151 (+ 289us) client_negotiation.cc:240] Negotiation successful
0808 15:58:05.826765 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":92,"thread_start_us":443,"threads_started":1}
I20260808 15:58:05.830848 9727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:58:05.832033 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.831543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:47994)
0808 15:58:05.831735 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.831872 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 15:58:05.832333 9677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
I20260808 15:58:05.833902 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.833574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48002)
0808 15:58:05.833719 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.833785 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 15:58:05.834187 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab.
Entry for principal kudu/127.27.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab.
I20260808 15:58:05.851967 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:39745
--webserver_interface=127.27.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab
--principal=kudu/127.27.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:41405,127.27.254.61:39745,127.27.254.60:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:58:05.883553 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } attempt: 1
I20260808 15:58:05.884788 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.884319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48014)
0808 15:58:05.884520 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.884652 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:58:05.885847 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.885517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48016)
0808 15:58:05.885654 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.885728 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 15:58:05.886078 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
I20260808 15:58:05.954329 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } attempt: 2
I20260808 15:58:05.955818 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.955131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48022)
0808 15:58:05.955501 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.955629 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 15:58:05.957041 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:05.956620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48030)
0808 15:58:05.956770 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:05.956861 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 15:58:05.957681 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
I20260808 15:58:06.049052 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } attempt: 3
I20260808 15:58:06.050264 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.049823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48044)
0808 15:58:06.050038 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.050140 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 15:58:06.051285 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.050963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48060)
0808 15:58:06.051090 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.051155 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:58:06.051576 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
W20260808 15:58:06.181135 9734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:06.181535 9734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:06.181694 9734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:06.188954 9734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:06.189062 9734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:06.189109 9734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:06.189196 9734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:06.189284 9734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:06.194870 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } attempt: 4
I20260808 15:58:06.196102 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.195562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48074)
0808 15:58:06.195873 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.195963 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260808 15:58:06.197181 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.196794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:48080)
0808 15:58:06.196982 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.197061 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 15:58:06.197453 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.27.254.61:39745: connect: Connection refused (error 111)
I20260808 15:58:06.200273 9734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:41405,127.27.254.61:39745,127.27.254.60:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab
--principal=kudu/127.27.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:06.201722 9734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:06.203006 9734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:06.214991 9740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:06.214960 9742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:06.215034 9739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:06.218730 9734 server_base.cc:1034] running on GCE node
I20260808 15:58:06.219521 9734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:06.220815 9734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:06.222013 9734 hybrid_clock.cc:648] HybridClock initialized: now 1786204686221977 us; error 30 us; skew 500 ppm
I20260808 15:58:06.222335 9734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:06.225373 9734 init.cc:377] Logged in from keytab as kudu/127.27.254.61@KRBTEST.COM (short username kudu)
I20260808 15:58:06.226861 9734 webserver.cc:460] Webserver started at http://127.27.254.61:38397/ using document root <none> and password file <none>
I20260808 15:58:06.227574 9734 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:06.227699 9734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:06.228003 9734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:06.230944 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data/instance:
uuid: "43272a90632b4b668af8e98986b17481"
format_stamp: "Formatted at 2026-08-08 15:58:06 on dist-test-slave-srgv"
server_key: "ad5fec617d5ba464d68b8a42dab1fac5"
server_key_iv: "ce9ad161f91994c377b566b84859df09"
server_key_version: "encryptionkey@0"
I20260808 15:58:06.231732 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal/instance:
uuid: "43272a90632b4b668af8e98986b17481"
format_stamp: "Formatted at 2026-08-08 15:58:06 on dist-test-slave-srgv"
server_key: "ad5fec617d5ba464d68b8a42dab1fac5"
server_key_iv: "ce9ad161f91994c377b566b84859df09"
server_key_version: "encryptionkey@0"
I20260808 15:58:06.236770 9734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:58:06.240044 9749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:06.241343 9734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:58:06.241565 9734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal
uuid: "43272a90632b4b668af8e98986b17481"
format_stamp: "Formatted at 2026-08-08 15:58:06 on dist-test-slave-srgv"
server_key: "ad5fec617d5ba464d68b8a42dab1fac5"
server_key_iv: "ce9ad161f91994c377b566b84859df09"
server_key_version: "encryptionkey@0"
I20260808 15:58:06.241753 9734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:06.256173 9734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:06.260155 9734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:06.260432 9734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:06.278733 9734 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.61:39745
I20260808 15:58:06.278774 9800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.61:39745 every 8 connection(s)
I20260808 15:58:06.280061 9734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
I20260808 15:58:06.283628 9801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:06.288640 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9734
I20260808 15:58:06.288844 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-1/wal/instance
I20260808 15:58:06.289366 28664 external_mini_cluster.cc:1428] Setting key 8775c64b57718e4efca1a068f09bd0ef
I20260808 15:58:06.290570 9801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.60@KRBTEST.COM; defaulting to no policy
I20260808 15:58:06.314147 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.294010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.27.254.62:41405)
0808 15:58:06.294236 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.294242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.295520 (+ 1278us) server_negotiation.cc:407] Connection header received
0808 15:58:06.298311 (+ 2791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.298315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.298385 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.298472 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.299885 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.301048 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.302637 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.303454 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.303604 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.307066 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.307074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.307081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.307127 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.310957 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.311658 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.311665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.311669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.311817 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.312224 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.312229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.312232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.312654 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.312811 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.313531 (+ 720us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.313782 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:58:06.314901 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.294438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41405 (local address 127.0.0.1:60294)
0808 15:58:06.295317 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.295369 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.298019 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.298714 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.298721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.299205 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.299726 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.299741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.301205 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.301208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.302458 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.302470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.303667 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.303671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.303848 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.304539 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.304559 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.306835 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.311109 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.311116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.311128 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.311524 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.311929 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.311932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.311934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.312095 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.312790 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.312795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.313097 (+ 302us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.313319 (+ 222us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.313751 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.60@KRBTEST.COM" created.
I20260808 15:58:06.316805 9801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 15:58:06.333231 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.317707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:34376)
0808 15:58:06.317938 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.317968 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.318076 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.318935 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.318943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.319433 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.319632 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.319645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.320940 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.320973 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.322509 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.322526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.323307 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.323319 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.323541 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.324154 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.324180 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.326579 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.330899 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.330902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.330905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.331259 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.331687 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.331690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.331692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.331816 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.332563 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.332567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.332689 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.332789 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.332982 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 15:58:06.333820 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.317842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.27.254.61:39745)
0808 15:58:06.318529 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.318535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.318558 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:58:06.318598 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.318604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.318765 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.318888 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.319795 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.320799 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.322642 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.323150 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.323243 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.326808 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.326825 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.326843 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.326886 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.330747 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.331411 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.331423 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.331429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.331580 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.331929 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.331937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.331942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.332416 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.332657 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.332997 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.333277 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":413,"threads_started":1}
I20260808 15:58:06.336170 9801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:58:06.337292 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.336861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39474)
0808 15:58:06.337032 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.337162 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 15:58:06.337620 9752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.339190 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.338856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39484)
0808 15:58:06.339028 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.339099 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 15:58:06.339397 9801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab.
Entry for principal kudu/127.27.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab.
I20260808 15:58:06.349203 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:44777
--webserver_interface=127.27.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab
--principal=kudu/127.27.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:41405,127.27.254.61:39745,127.27.254.60:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:58:06.388711 9801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 1
I20260808 15:58:06.390072 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.389505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39490)
0808 15:58:06.389719 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.389874 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 15:58:06.391121 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.390693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39494)
0808 15:58:06.390857 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.390973 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:58:06.391322 9801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.459560 9801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 2
I20260808 15:58:06.460824 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.460284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39500)
0808 15:58:06.460516 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.460632 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 15:58:06.461900 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.461535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39506)
0808 15:58:06.461678 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.461770 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 15:58:06.462116 9801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.496722 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } attempt: 5
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 15:58:06.510030 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.497502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:34378)
0808 15:58:06.497703 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.497725 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.497869 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.498253 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.498260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.498665 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.498856 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.498865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.499851 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.499859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.500698 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.500709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.501435 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.501441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.501618 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.502353 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.502371 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.504611 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.507597 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.507604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.507607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.508006 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.508379 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.508384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.508387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.508523 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.509020 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.509025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.509196 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.509333 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.509640 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 15:58:06.510030 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.497605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.27.254.61:39745)
0808 15:58:06.497824 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.497830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.497853 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:58:06.498013 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.498018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.498102 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.498210 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.499015 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.499703 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.500844 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.501277 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.501406 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.504803 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.504808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.504811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.504844 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.507462 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.508131 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.508138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.508141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.508260 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.508631 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.508636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.508639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.508905 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.509051 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.509463 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.509667 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 15:58:06.511202 9727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:58:06.512187 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.511837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39522)
0808 15:58:06.511987 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.512068 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:58:06.513116 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.512832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39528)
0808 15:58:06.512954 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.513029 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 15:58:06.513339 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.553357 9801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 3
I20260808 15:58:06.554701 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.554050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39534)
0808 15:58:06.554314 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.554467 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 15:58:06.555977 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.555433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39540)
0808 15:58:06.555697 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.555809 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260808 15:58:06.556231 9801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.564589 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 1
I20260808 15:58:06.565676 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.565212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39542)
0808 15:58:06.565385 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.565526 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 15:58:06.566599 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.566285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39546)
0808 15:58:06.566431 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.566504 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:58:06.566813 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.635015 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 2
I20260808 15:58:06.636124 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.635655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39550)
0808 15:58:06.635875 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.635967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:58:06.637033 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.636704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39562)
0808 15:58:06.636871 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.636939 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 15:58:06.637235 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
W20260808 15:58:06.676494 9808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:06.676818 9808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:06.676982 9808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:06.684033 9808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:06.684166 9808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:06.684265 9808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:06.684360 9808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:06.684440 9808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:06.694048 9808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:41405,127.27.254.61:39745,127.27.254.60:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab
--principal=kudu/127.27.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:06.695374 9808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:06.696642 9808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 15:58:06.699474 9801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 4
I20260808 15:58:06.700578 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.700118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39574)
0808 15:58:06.700351 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.700433 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 15:58:06.701661 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.701238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39576)
0808 15:58:06.701385 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.701523 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:58:06.701902 9801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
W20260808 15:58:06.708617 9813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:06.708690 9816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:06.708722 9814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:06.709486 9808 server_base.cc:1034] running on GCE node
I20260808 15:58:06.710224 9808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:06.711503 9808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:06.712699 9808 hybrid_clock.cc:648] HybridClock initialized: now 1786204686712654 us; error 40 us; skew 500 ppm
I20260808 15:58:06.713011 9808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:06.716033 9808 init.cc:377] Logged in from keytab as kudu/127.27.254.60@KRBTEST.COM (short username kudu)
I20260808 15:58:06.717479 9808 webserver.cc:460] Webserver started at http://127.27.254.60:33659/ using document root <none> and password file <none>
I20260808 15:58:06.718111 9808 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:06.718214 9808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:06.718444 9808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:06.721204 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data/instance:
uuid: "632106ab93284026aa288f9264d4709e"
format_stamp: "Formatted at 2026-08-08 15:58:06 on dist-test-slave-srgv"
server_key: "f4209bd5aeb53debcb080df5ea5216d1"
server_key_iv: "16b13c6bd0f94fd362f5e7d20905830d"
server_key_version: "encryptionkey@0"
I20260808 15:58:06.722009 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal/instance:
uuid: "632106ab93284026aa288f9264d4709e"
format_stamp: "Formatted at 2026-08-08 15:58:06 on dist-test-slave-srgv"
server_key: "f4209bd5aeb53debcb080df5ea5216d1"
server_key_iv: "16b13c6bd0f94fd362f5e7d20905830d"
server_key_version: "encryptionkey@0"
I20260808 15:58:06.726994 9808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:58:06.730410 9823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:06.731678 9808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:58:06.731813 9808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal
uuid: "632106ab93284026aa288f9264d4709e"
format_stamp: "Formatted at 2026-08-08 15:58:06 on dist-test-slave-srgv"
server_key: "f4209bd5aeb53debcb080df5ea5216d1"
server_key_iv: "16b13c6bd0f94fd362f5e7d20905830d"
server_key_version: "encryptionkey@0"
I20260808 15:58:06.731984 9808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:06.743438 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 3
I20260808 15:58:06.744529 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.744054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39578)
0808 15:58:06.744271 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.744356 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:58:06.745669 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.745233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39590)
0808 15:58:06.745452 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.745536 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260808 15:58:06.745893 9727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:44777: Network error: Client connection negotiation failed: client connection to 127.27.254.60:44777: connect: Connection refused (error 111)
I20260808 15:58:06.750614 9808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:06.754796 9808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:06.755028 9808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:06.773522 9808 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.60:44777
I20260808 15:58:06.773545 9874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.60:44777 every 8 connection(s)
I20260808 15:58:06.774912 9808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
I20260808 15:58:06.778643 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:06.782250 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9808
I20260808 15:58:06.782457 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204485174398-28664-0/minicluster-data/master-2/wal/instance
I20260808 15:58:06.783077 28664 external_mini_cluster.cc:1428] Setting key de0ab1ff849f17c1e12227dfc0783cfb
I20260808 15:58:06.786283 9875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:06.805922 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.790042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.27.254.62:41405)
0808 15:58:06.790279 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.790286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.791132 (+ 846us) server_negotiation.cc:407] Connection header received
0808 15:58:06.791441 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.791448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.791519 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.791631 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.793015 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.794077 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.795809 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.796268 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.796401 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.799993 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.799997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.800000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.800030 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.802896 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.803604 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.803609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.803613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.803760 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.804133 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.804139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.804142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.804390 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.804551 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.805264 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.805533 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:58:06.806424 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.784919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.27.254.60:44777)
0808 15:58:06.785675 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.785687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.785727 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:58:06.787341 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.787364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.787768 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.788143 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.789965 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.791790 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.793660 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.794432 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.794621 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.799633 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.799642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.799656 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.799691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.802454 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.803166 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.803173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.803175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.803328 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.803740 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.803744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.803746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.804194 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.804498 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.804878 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.805303 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
I20260808 15:58:06.806464 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.790085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:41405 (local address 127.0.0.1:60302)
0808 15:58:06.790986 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.791039 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.791301 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.791687 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.791694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.792471 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.792853 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.792866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.794245 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.794251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.795664 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.795675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.796500 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.796507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.796658 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.797223 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.797243 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.799790 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.803044 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.803054 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.803065 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.803477 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.803890 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.803895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.803898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.804027 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.804515 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.804524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.804716 (+ 192us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.804974 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.805493 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":162,"spinlock_wait_cycles":14208,"thread_start_us":439,"threads_started":1}
I20260808 15:58:06.808130 9875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:06.822108 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.809091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:39745 (local address 127.0.0.1:34384)
0808 15:58:06.809260 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.809278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.809413 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.809992 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.809997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.810453 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.810652 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.810666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.811677 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.811681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.812664 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.812678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.813335 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.813341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.813579 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.814181 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.814197 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.816405 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.819769 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.819773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.819775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.820146 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.820605 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.820609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.820611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.820735 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.821310 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.821313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.821491 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.821602 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.821792 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:58:06.822439 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.809014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.27.254.61:39745)
0808 15:58:06.809219 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.809227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.809362 (+ 135us) server_negotiation.cc:407] Connection header received
0808 15:58:06.809580 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.809585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.809672 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.809783 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.810809 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.811510 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.812803 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.813199 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.813290 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.816588 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.816593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.816596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.816627 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.819648 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.820280 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.820295 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.820299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.820489 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.820846 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.820851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.820854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.821195 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.821350 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.821725 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.821944 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:58:06.823428 9875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:58:06.824815 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.809626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.27.254.62:41405)
0808 15:58:06.809782 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.809786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.809977 (+ 191us) server_negotiation.cc:407] Connection header received
0808 15:58:06.810115 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.810119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.810185 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.810271 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.811599 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.812320 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.813972 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.814420 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.814510 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.818706 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.818712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.818717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.818758 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.821850 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.822720 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.822724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.822727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.822843 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.823246 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.823250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.823252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.823524 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.823782 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.824276 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.824494 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 15:58:06.836592 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.824177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39620)
0808 15:58:06.824406 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.824419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.824538 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.824810 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.824814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.825285 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.825482 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.825493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.826610 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.826615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.827413 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.827421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.828366 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.828370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.828527 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.829078 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.829094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.831007 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.833897 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.833902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.833906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.834238 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.834682 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.834685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.834687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.834802 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.835390 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.835396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.835578 (+ 182us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.835732 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.836010 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260808 15:58:06.836634 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.824165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.27.254.60:44777)
0808 15:58:06.824337 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.824343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.824480 (+ 137us) server_negotiation.cc:407] Connection header received
0808 15:58:06.824644 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.824649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.824706 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.824795 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.825608 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.826471 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.827565 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.828233 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.828372 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.831189 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.831194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.831197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.831221 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.833733 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.834397 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.834402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.834407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.834568 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.834911 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.834917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.834920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.835289 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.835424 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.835811 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.836209 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":63}
I20260808 15:58:06.840981 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.826501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.27.254.61:39745)
0808 15:58:06.826650 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.826658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.826675 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:58:06.826853 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.826858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.826929 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.827031 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.828520 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.829183 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.830528 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.830972 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.831116 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.835468 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.835474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.835479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.835517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.838519 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.839255 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.839258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.839260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.839397 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.839798 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.839801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.839803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.840048 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.840233 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.840557 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.840727 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:58:06.842437 9875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: Bootstrap starting.
I20260808 15:58:06.845921 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:58:06.846388 9875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:06.847604 9875 log.cc:824] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:06.850834 9875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: No bootstrap required, opened a new log
I20260808 15:58:06.864763 9875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:06.865336 9875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:06.865536 9875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632106ab93284026aa288f9264d4709e, State: Initialized, Role: FOLLOWER
I20260808 15:58:06.866257 9875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:06.867805 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:06.868378 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:06.869125 9875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:06.873911 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.852030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.27.254.61:39745)
0808 15:58:06.852176 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.852182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.854412 (+ 2230us) server_negotiation.cc:407] Connection header received
0808 15:58:06.854624 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.854630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.854710 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.854808 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.857304 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.858478 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.861032 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.861761 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.861893 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.866083 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.866092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.866099 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.866154 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.870023 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.871065 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.871071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.871074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.871241 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.871784 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.871792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.871797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.872235 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.872471 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.873179 (+ 708us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.873459 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:58:06.884429 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.851580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.27.254.62:41405)
0808 15:58:06.851760 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.851768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.859809 (+ 8041us) server_negotiation.cc:407] Connection header received
0808 15:58:06.860056 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.860061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.860141 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.860207 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.865804 (+ 5597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.866824 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.868824 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.869493 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.869590 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.877028 (+ 7438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.877036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.877044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.877098 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.880801 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.881754 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.881759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.881762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.881943 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.882463 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.882470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.882476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.882968 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.883265 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.883812 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.884074 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 15:58:06.886204 9899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:58:06.886411 9899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:58:06.888561 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.852135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.27.254.60:44777)
0808 15:58:06.852314 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.852322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.855557 (+ 3235us) server_negotiation.cc:407] Connection header received
0808 15:58:06.856093 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.856101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.856191 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.856267 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.862560 (+ 6293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.863682 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.870616 (+ 6934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.872778 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.872891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.881335 (+ 8444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.881341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.881346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.881401 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.885206 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.886082 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.886087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.886090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.886268 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.886725 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.886750 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.886753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.887060 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.887372 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.887647 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.887916 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:58:06.881757 9808 master_runner.cc:186] Error getting master registration for 127.27.254.62:41405: OK, instance_id { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" instance_seqno: 1786204685786719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:58:06.894668 9808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:58:06.923096 9727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 4
Aug 08 15:58:06 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204685, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
I20260808 15:58:06.937296 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.923693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39628)
0808 15:58:06.923897 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:06.923929 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:06.924072 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:06.924415 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:06.924420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:06.924812 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:06.924978 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.924986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.926203 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.926210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.927530 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:06.927546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.928564 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.928570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:06.928748 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.929382 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:06.929399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:06.931469 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:06.934644 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.934651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.934655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.935116 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.935558 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:06.935563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:06.935567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:06.935708 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:06.936315 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:06.936320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:06.936481 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 15:58:06.936633 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 15:58:06.936905 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 15:58:06.937297 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:06.923779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.27.254.60:44777)
0808 15:58:06.923949 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:06.923954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:06.924013 (+ 59us) server_negotiation.cc:407] Connection header received
0808 15:58:06.924201 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:06.924212 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:06.924288 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:06.924385 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:06.925141 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.926058 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.927704 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:06.928410 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:06.928562 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:06.931653 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:06.931658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:06.931661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:06.931690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:06.934508 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.935245 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.935254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.935257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.935447 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:06.935819 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:06.935825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:06.935828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:06.936183 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:06.936335 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:06.936724 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 15:58:06.936944 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:58:06.942396 9727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: Bootstrap starting.
I20260808 15:58:06.945752 9727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:06.946744 9727 log.cc:824] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:06.949630 9727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: No bootstrap required, opened a new log
I20260808 15:58:06.957368 9727 raft_consensus.cc:348] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:06.957721 9727 raft_consensus.cc:374] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:06.957888 9727 raft_consensus.cc:729] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e509965a59a345f29d4fd2ec6387eb3f, State: Initialized, Role: FOLLOWER
I20260808 15:58:06.958416 9727 consensus_queue.cc:260] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:06.959455 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:06.959862 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:06.960537 9727 sys_catalog.cc:564] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:58:06.974745 9913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:58:06.974961 9913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:58:06.973733 9660 master_runner.cc:186] Error getting master registration for 127.27.254.61:39745: OK, instance_id { permanent_uuid: "43272a90632b4b668af8e98986b17481" instance_seqno: 1786204686273750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:58:06.985075 9660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:58:07.001190 9801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } attempt: 5
W20260808 15:58:07.007030 9829 tablet.cc:2369] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 15:58:07 dist-test-slave-srgv krb5kdc[9647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
I20260808 15:58:07.018880 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:07.001866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:44777 (local address 127.0.0.1:39638)
0808 15:58:07.002166 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:07.002186 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:07.002311 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:07.002698 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:07.002704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:07.003256 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:07.003468 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:07.003482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:07.004966 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:07.004974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:07.006438 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:07.006453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:07.007669 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:07.007677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:07.007919 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:07.008624 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:07.008651 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:07.011750 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:07.016020 (+ 4270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:07.016028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:07.016032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:07.016601 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:07.017020 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:07.017025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:07.017030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:07.017200 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:07.017846 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:07.017852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:07.018010 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 15:58:07.018181 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 15:58:07.018489 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260808 15:58:07.018954 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:07.001989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.27.254.60:44777)
0808 15:58:07.002170 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:07.002177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:07.002271 (+ 94us) server_negotiation.cc:407] Connection header received
0808 15:58:07.002441 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:07.002447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:07.002529 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:07.002643 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:07.003633 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:07.004786 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:07.006604 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:07.007299 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:07.007437 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:07.011988 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:07.011995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:07.012000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:07.012038 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:07.015871 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:07.016736 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:07.016742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:07.016746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:07.016899 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:07.017325 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:07.017331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:07.017334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:07.017715 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:07.017889 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:07.018301 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 15:58:07.018555 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:58:07.025346 9801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: Bootstrap starting.
W20260808 15:58:07.026938 9681 tablet.cc:2369] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:07.031008 9801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:07.032300 9801 log.cc:824] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:07.035588 9801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: No bootstrap required, opened a new log
I20260808 15:58:07.046387 9801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:07.046777 9801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:07.046888 9801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43272a90632b4b668af8e98986b17481, State: Initialized, Role: FOLLOWER
I20260808 15:58:07.047516 9801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:07.050241 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.050645 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:07.052107 9801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:07.066802 9734 master_runner.cc:263] No leader master found from master 43272a90632b4b668af8e98986b17481
I20260808 15:58:07.066967 9734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 15:58:07.072136 9927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:58:07.072361 9927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:58:07.192670 9888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:07.193027 9888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:07.194670 9888 leader_election.cc:288] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e509965a59a345f29d4fd2ec6387eb3f (127.27.254.62:41405), 43272a90632b4b668af8e98986b17481 (127.27.254.61:39745)
I20260808 15:58:07.196060 9702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632106ab93284026aa288f9264d4709e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e509965a59a345f29d4fd2ec6387eb3f" is_pre_election: true
I20260808 15:58:07.196652 9702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632106ab93284026aa288f9264d4709e in term 0.
I20260808 15:58:07.197317 9827 leader_election.cc:302] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 632106ab93284026aa288f9264d4709e, e509965a59a345f29d4fd2ec6387eb3f; no voters:
I20260808 15:58:07.197896 9888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:58:07.198048 9888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:58:07.197803 9776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632106ab93284026aa288f9264d4709e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43272a90632b4b668af8e98986b17481" is_pre_election: true
I20260808 15:58:07.198158 9888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:07.198477 9776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632106ab93284026aa288f9264d4709e in term 0.
I20260808 15:58:07.202669 9888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:07.203773 9702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632106ab93284026aa288f9264d4709e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e509965a59a345f29d4fd2ec6387eb3f"
I20260808 15:58:07.203990 9702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:07.207489 9702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632106ab93284026aa288f9264d4709e in term 1.
I20260808 15:58:07.208022 9827 leader_election.cc:302] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 632106ab93284026aa288f9264d4709e, e509965a59a345f29d4fd2ec6387eb3f; no voters:
I20260808 15:58:07.209846 9776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632106ab93284026aa288f9264d4709e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43272a90632b4b668af8e98986b17481"
I20260808 15:58:07.210080 9776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:07.213537 9776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632106ab93284026aa288f9264d4709e in term 1.
I20260808 15:58:07.214325 9888 leader_election.cc:288] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [CANDIDATE]: Term 1 election: Requested vote from peers e509965a59a345f29d4fd2ec6387eb3f (127.27.254.62:41405), 43272a90632b4b668af8e98986b17481 (127.27.254.61:39745)
I20260808 15:58:07.214735 9888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:07.215176 9888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [term 1 LEADER]: Becoming Leader. State: Replica: 632106ab93284026aa288f9264d4709e, State: Running, Role: LEADER
I20260808 15:58:07.216001 9888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } }
I20260808 15:58:07.226193 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632106ab93284026aa288f9264d4709e. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.230630 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:07.241948 9931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:07.243958 9931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:07.258997 9702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [term 1 FOLLOWER]: Refusing update from remote peer 632106ab93284026aa288f9264d4709e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:07.260099 9776 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [term 1 FOLLOWER]: Refusing update from remote peer 632106ab93284026aa288f9264d4709e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:07.260896 9888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:07.261240 9929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:58:07.266086 9755 tablet.cc:2369] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:07.291396 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632106ab93284026aa288f9264d4709e. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.291827 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:07.295387 9932 mvcc.cc:205] Tried to move back new op lower bound from 7316294399001796608 to 7316294398849511424. Current Snapshot: MvccSnapshot[applied={T|T < 7316294399001796608}]
I20260808 15:58:07.306613 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632106ab93284026aa288f9264d4709e. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.306968 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:07.310998 9934 mvcc.cc:205] Tried to move back new op lower bound from 7316294399001796608 to 7316294398849511424. Current Snapshot: MvccSnapshot[applied={T|T < 7316294399001796608}]
I20260808 15:58:07.312920 9902 sys_catalog.cc:455] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.313181 9902 sys_catalog.cc:458] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:07.313671 9929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.314009 9929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:07.314373 9929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.314616 9929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:07.315659 9931 catalog_manager.cc:1347] Generated new cluster ID: 7be90a2c33b84576812b500d2b834836
I20260808 15:58:07.315752 9931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:58:07.327749 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "632106ab93284026aa288f9264d4709e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e509965a59a345f29d4fd2ec6387eb3f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41405 } } peers { permanent_uuid: "43272a90632b4b668af8e98986b17481" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 39745 } } peers { permanent_uuid: "632106ab93284026aa288f9264d4709e" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 44777 } } }
I20260808 15:58:07.328051 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:58:07.343142 9931 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:58:07.345389 9931 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:58:07.365757 9931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 632106ab93284026aa288f9264d4709e: Generated new TSK 0
I20260808 15:58:07.371244 9931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:58:07.976974 9913 catalog_manager.cc:1259] Loaded cluster ID: 7be90a2c33b84576812b500d2b834836
I20260808 15:58:07.977108 9913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: loading cluster ID for follower catalog manager: success
I20260808 15:58:07.979607 9913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: acquiring CA information for follower catalog manager: success
I20260808 15:58:07.981320 9913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e509965a59a345f29d4fd2ec6387eb3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:58:08.074414 9927 catalog_manager.cc:1259] Loaded cluster ID: 7be90a2c33b84576812b500d2b834836
I20260808 15:58:08.074513 9927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: loading cluster ID for follower catalog manager: success
I20260808 15:58:08.077049 9927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: acquiring CA information for follower catalog manager: success
I20260808 15:58:08.078966 9927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43272a90632b4b668af8e98986b17481: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:58:08.506858 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9660
I20260808 15:58:08.508183 9722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:08.619768 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9660
I20260808 15:58:08.640794 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9734
I20260808 15:58:08.641918 9796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:08.747346 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9734
I20260808 15:58:08.767493 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9808
I20260808 15:58:08.768535 9870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:08.875690 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9808
2026-08-08T15:58:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4838 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:58:08 dist-test-slave-srgv krb5kdc[9951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:58:08 dist-test-slave-srgv krb5kdc[9951](info): set up 2 sockets
Aug 08 15:58:08 dist-test-slave-srgv krb5kdc[9951](info): commencing operation
krb5kdc: starting...
W20260808 15:58:09.975970 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:58:10 dist-test-slave-srgv krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:58:10.212484 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46101
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:10.532775 9964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:10.533064 9964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:10.533228 9964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:10.540515 9964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:10.540623 9964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:10.540725 9964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:10.540828 9964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:10.540911 9964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:10.550683 9964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:46101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:10.551994 9964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:10.553333 9964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:10.565426 9969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:10.565502 9970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:10.565522 9972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:10.566313 9964 server_base.cc:1034] running on GCE node
I20260808 15:58:10.567224 9964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:10.568572 9964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:10.569785 9964 hybrid_clock.cc:648] HybridClock initialized: now 1786204690569729 us; error 29 us; skew 500 ppm
I20260808 15:58:10.570091 9964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:10 dist-test-slave-srgv krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:10.573196 9964 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:10.574784 9964 webserver.cc:460] Webserver started at http://127.27.254.62:35863/ using document root <none> and password file <none>
I20260808 15:58:10.575470 9964 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:10.575596 9964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:10.575896 9964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:10.578877 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "01e26e0f301542f8a5caa7392f39acdc"
format_stamp: "Formatted at 2026-08-08 15:58:10 on dist-test-slave-srgv"
server_key: "1958911af5001a52ded61b9d71dca834"
server_key_iv: "d0dd7e4aa38ffe3edaa8b125d43fea03"
server_key_version: "encryptionkey@0"
I20260808 15:58:10.579635 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "01e26e0f301542f8a5caa7392f39acdc"
format_stamp: "Formatted at 2026-08-08 15:58:10 on dist-test-slave-srgv"
server_key: "1958911af5001a52ded61b9d71dca834"
server_key_iv: "d0dd7e4aa38ffe3edaa8b125d43fea03"
server_key_version: "encryptionkey@0"
I20260808 15:58:10.584817 9964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:58:10.588306 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:10.589671 9964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:10.589876 9964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "01e26e0f301542f8a5caa7392f39acdc"
format_stamp: "Formatted at 2026-08-08 15:58:10 on dist-test-slave-srgv"
server_key: "1958911af5001a52ded61b9d71dca834"
server_key_iv: "d0dd7e4aa38ffe3edaa8b125d43fea03"
server_key_version: "encryptionkey@0"
I20260808 15:58:10.590056 9964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:10.613276 9964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:10.617879 9964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:10.618198 9964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:10.636960 9964 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:46101
I20260808 15:58:10.636979 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:46101 every 8 connection(s)
I20260808 15:58:10.638492 9964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:10.642238 10031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:10.645385 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 9964
I20260808 15:58:10.645630 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:58:10.646517 28664 external_mini_cluster.cc:1428] Setting key 3372bb30df2a3078f4fc31b75bf6821e
I20260808 15:58:10.655138 10031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc: Bootstrap starting.
I20260808 15:58:10.658999 10031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:10.660348 10031 log.cc:824] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:58:10 dist-test-slave-srgv krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:10.663368 10031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc: No bootstrap required, opened a new log
I20260808 15:58:10.670528 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:10.648466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.27.254.62:46101)
0808 15:58:10.649255 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:10.649267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:10.649306 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:58:10.651035 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:10.651056 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:10.651457 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:10.651782 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:10.653656 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:10.654936 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:10.656858 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:10.657603 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:10.657739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:10.662987 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:10.662995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:10.663010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:10.663047 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:10.666260 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:10.667106 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:10.667116 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:10.667126 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:10.667350 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:10.667802 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:10.667805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:10.667807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:10.668329 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:10.668548 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:10.669058 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 15:58:10.669417 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":370,"threads_started":1}
I20260808 15:58:10.672004 10031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e26e0f301542f8a5caa7392f39acdc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46101 } }
I20260808 15:58:10.672382 10031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:10.672502 10031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e26e0f301542f8a5caa7392f39acdc, State: Initialized, Role: FOLLOWER
I20260808 15:58:10.673148 10031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e26e0f301542f8a5caa7392f39acdc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46101 } }
I20260808 15:58:10.673494 10031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:58:10.673655 10031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:58:10.673852 10031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:10.674513 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:58:10.676149 10031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e26e0f301542f8a5caa7392f39acdc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46101 } }
I20260808 15:58:10.676729 10031 leader_election.cc:302] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e26e0f301542f8a5caa7392f39acdc; no voters:
I20260808 15:58:10.677368 10031 leader_election.cc:288] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:58:10.677536 10036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:10.678289 10036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [term 1 LEADER]: Becoming Leader. State: Replica: 01e26e0f301542f8a5caa7392f39acdc, State: Running, Role: LEADER
I20260808 15:58:10.678884 10036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e26e0f301542f8a5caa7392f39acdc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46101 } }
I20260808 15:58:10.679490 10031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:10.681703 10037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e26e0f301542f8a5caa7392f39acdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e26e0f301542f8a5caa7392f39acdc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46101 } } }
I20260808 15:58:10.682173 10037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:10.682966 10038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e26e0f301542f8a5caa7392f39acdc. Latest consensus state: current_term: 1 leader_uuid: "01e26e0f301542f8a5caa7392f39acdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e26e0f301542f8a5caa7392f39acdc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46101 } } }
I20260808 15:58:10.683324 10038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:10.686337 10043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:10.691313 10043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:10.700322 10043 catalog_manager.cc:1347] Generated new cluster ID: bcbeddae74aa46dd9c8d21716b457716
I20260808 15:58:10.700452 10043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:58:10.716403 10043 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:58:10.717787 10043 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:58:10.726711 10043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e26e0f301542f8a5caa7392f39acdc: Generated new TSK 0
I20260808 15:58:10.727500 10043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:58:10.734743 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 9964
I20260808 15:58:10.735893 10026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:10.824744 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 9964
2026-08-08T15:58:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1947 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:58:10 dist-test-slave-srgv krb5kdc[10059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:58:10 dist-test-slave-srgv krb5kdc[10059](info): set up 2 sockets
Aug 08 15:58:10 dist-test-slave-srgv krb5kdc[10059](info): commencing operation
krb5kdc: starting...
W20260808 15:58:11.908015 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:58:12 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:58:12.149091 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36521
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:12.467796 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:12.468070 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:12.468154 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:12.475462 10072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:12.475561 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:12.475608 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:12.475663 10072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:12.475704 10072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:12.485081 10072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:12.486303 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:12.487480 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:12.501133 10081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:12.501201 10078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:12.501176 10079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:12.501870 10072 server_base.cc:1034] running on GCE node
I20260808 15:58:12.502622 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:12.503930 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:12.505127 10072 hybrid_clock.cc:648] HybridClock initialized: now 1786204692505084 us; error 33 us; skew 500 ppm
I20260808 15:58:12.505548 10072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:12 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:12.508524 10072 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:12.510056 10072 webserver.cc:460] Webserver started at http://127.27.254.62:36819/ using document root <none> and password file <none>
I20260808 15:58:12.510722 10072 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:12.510870 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:12.511161 10072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:12.514056 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "9c2f3b6d642549a1b7e84c8d1abf89da"
format_stamp: "Formatted at 2026-08-08 15:58:12 on dist-test-slave-srgv"
server_key: "54225eba4446fca2d18e594ea2e07034"
server_key_iv: "824ae82b08600bc55ab6c1783f972e76"
server_key_version: "encryptionkey@0"
I20260808 15:58:12.514792 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "9c2f3b6d642549a1b7e84c8d1abf89da"
format_stamp: "Formatted at 2026-08-08 15:58:12 on dist-test-slave-srgv"
server_key: "54225eba4446fca2d18e594ea2e07034"
server_key_iv: "824ae82b08600bc55ab6c1783f972e76"
server_key_version: "encryptionkey@0"
I20260808 15:58:12.519747 10072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:58:12.523078 10088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:12.524327 10072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:58:12.524518 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "9c2f3b6d642549a1b7e84c8d1abf89da"
format_stamp: "Formatted at 2026-08-08 15:58:12 on dist-test-slave-srgv"
server_key: "54225eba4446fca2d18e594ea2e07034"
server_key_iv: "824ae82b08600bc55ab6c1783f972e76"
server_key_version: "encryptionkey@0"
I20260808 15:58:12.524713 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:12.540151 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:12.543972 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:12.544250 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:12.562273 10072 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:36521
I20260808 15:58:12.562302 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:36521 every 8 connection(s)
I20260808 15:58:12.563606 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:12.567219 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:12.572266 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10072
I20260808 15:58:12.572504 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:58:12.573371 28664 external_mini_cluster.cc:1428] Setting key 7e0874906e6cd688fba4736488ca5a1e
I20260808 15:58:12.579080 10140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap starting.
I20260808 15:58:12.582785 10140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:12.584021 10140 log.cc:824] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:12.586795 10140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: No bootstrap required, opened a new log
Aug 08 15:58:12 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:12.594326 10140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:12.594700 10140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:12.594821 10140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2f3b6d642549a1b7e84c8d1abf89da, State: Initialized, Role: FOLLOWER
I20260808 15:58:12.595463 10140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:12.595706 10140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:58:12.595865 10140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:58:12.596063 10140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:12.596628 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:12.575425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.27.254.62:36521)
0808 15:58:12.576264 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:12.576276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:12.576318 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:58:12.578096 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:12.578118 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:12.578486 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:12.578814 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:12.580578 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:12.581955 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:12.583830 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:12.584661 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:12.584862 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:12.588968 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:12.588976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:12.588991 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:12.589033 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:12.592265 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:12.592996 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:12.593002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:12.593008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:12.593190 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:12.593654 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:12.593659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:12.593662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:12.594182 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:12.594524 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:12.594992 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 15:58:12.595461 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":407,"threads_started":1}
I20260808 15:58:12.598917 10140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:12.599512 10140 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2f3b6d642549a1b7e84c8d1abf89da; no voters:
I20260808 15:58:12.600093 10140 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:58:12.600229 10145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:12.600904 10145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 LEADER]: Becoming Leader. State: Replica: 9c2f3b6d642549a1b7e84c8d1abf89da, State: Running, Role: LEADER
I20260808 15:58:12.601511 10145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:12.602183 10140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:12.604736 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2f3b6d642549a1b7e84c8d1abf89da. Latest consensus state: current_term: 1 leader_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } } }
I20260808 15:58:12.604707 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } } }
I20260808 15:58:12.605238 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:12.605238 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: This master's current role is: LEADER
W20260808 15:58:12.613482 10162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:58:12.613597 10162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:58:12.613760 10154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:12.615535 10154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:12.623605 10154 catalog_manager.cc:1347] Generated new cluster ID: cd43036ac2a44d01994f8453636005d9
I20260808 15:58:12.623720 10154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:58:12.632268 10154 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:58:12.633572 10154 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:58:12.640760 10154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Generated new TSK 0
I20260808 15:58:12.641757 10154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:58:12.711555 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:36521
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:13.042990 10167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:13.043316 10167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:13.043481 10167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:13.050429 10167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:13.050582 10167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:13.050747 10167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:58:13.059876 10167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:13.061123 10167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:13.062472 10167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:13.075708 10172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:13.075734 10175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:13.075668 10173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:13.076723 10167 server_base.cc:1034] running on GCE node
I20260808 15:58:13.077297 10167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:13.078322 10167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:13.079524 10167 hybrid_clock.cc:648] HybridClock initialized: now 1786204693079491 us; error 30 us; skew 500 ppm
I20260808 15:58:13.079787 10167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:13 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:13.082904 10167 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:58:13.084259 10167 webserver.cc:460] Webserver started at http://127.27.254.1:36201/ using document root <none> and password file <none>
I20260808 15:58:13.084874 10167 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:13.084972 10167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:13.085238 10167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:13.088055 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "19db8403f67d4adf35834ba08472f404"
server_key_iv: "48c5cad7cc593a091b365b570f27ff56"
server_key_version: "encryptionkey@0"
I20260808 15:58:13.088737 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "19db8403f67d4adf35834ba08472f404"
server_key_iv: "48c5cad7cc593a091b365b570f27ff56"
server_key_version: "encryptionkey@0"
I20260808 15:58:13.093780 10167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260808 15:58:13.097163 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:13.098501 10167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:13.098634 10167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "19db8403f67d4adf35834ba08472f404"
server_key_iv: "48c5cad7cc593a091b365b570f27ff56"
server_key_version: "encryptionkey@0"
I20260808 15:58:13.098795 10167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:13.114216 10167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:13.117770 10167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:13.118027 10167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:13.118845 10167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:13.120020 10167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:13.120103 10167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:13.120198 10167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:13.120242 10167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:13.152765 10167 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:37009
I20260808 15:58:13.152841 10295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:37009 every 8 connection(s)
I20260808 15:58:13.154114 10167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:58:13.161701 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10167
I20260808 15:58:13.161924 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:58:13.162639 28664 external_mini_cluster.cc:1428] Setting key 33f1ae29dc5760f51fa9618aae58de2e
Aug 08 15:58:13 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:13.174739 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:13.156515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:54785 (local address 127.27.254.62:36521)
0808 15:58:13.156937 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:13.156942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:13.157942 (+ 1000us) server_negotiation.cc:407] Connection header received
0808 15:58:13.159646 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:13.159650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:13.159715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:13.159811 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:13.161144 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.162284 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.164013 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.164495 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.164636 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:13.167887 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:13.167903 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:13.167910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:13.167954 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:13.171534 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:13.172205 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:13.172212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:13.172219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:13.172366 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:13.172851 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:13.172859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:13.172866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:13.173211 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:13.173324 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:13.174311 (+ 987us) server_negotiation.cc:299] Negotiation successful
0808 15:58:13.174483 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":210,"threads_started":1}
I20260808 15:58:13.176127 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:13.157063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:54785)
0808 15:58:13.157777 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:13.157825 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:13.159407 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:13.160020 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:13.160027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:13.160487 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:13.161005 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:13.161021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.162473 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.162477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:13.163880 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:13.163892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.164712 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.164717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:13.164915 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:13.165566 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:13.165586 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:13.167666 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:13.171678 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:13.171683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:13.171695 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:13.172088 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:13.172476 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:13.172479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:13.172481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:13.172742 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:13.173345 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:13.173355 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:13.173824 (+ 469us) client_negotiation.cc:769] Sending connection context
0808 15:58:13.174156 (+ 332us) client_negotiation.cc:240] Negotiation successful
0808 15:58:13.174730 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260808 15:58:13.177695 10296 heartbeater.cc:347] Connected to a master server at 127.27.254.62:36521
I20260808 15:58:13.177994 10296 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:13.178675 10296 heartbeater.cc:511] Master 127.27.254.62:36521 requested a full tablet report, sending...
I20260808 15:58:13.180531 10105 ts_manager.cc:194] Registered new tserver with Master: 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1:37009)
I20260808 15:58:13.182053 10105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:54785
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:58:13.216536 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:36521
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:13.536342 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:13.536597 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:13.536741 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:13.543867 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:13.543987 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:13.544168 10302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:58:13.553534 10302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:13.554744 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:13.556043 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:13.568787 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:13.568812 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:13.569379 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:13.569789 10302 server_base.cc:1034] running on GCE node
I20260808 15:58:13.570451 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:13.571504 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:13.572723 10302 hybrid_clock.cc:648] HybridClock initialized: now 1786204693572667 us; error 54 us; skew 500 ppm
I20260808 15:58:13.573047 10302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:13 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:13.576637 10302 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:58:13.578088 10302 webserver.cc:460] Webserver started at http://127.27.254.2:33415/ using document root <none> and password file <none>
I20260808 15:58:13.578728 10302 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:13.578866 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:13.579164 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:13.581997 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "25ed9653d16a4dcba9808d07e4ac8c99"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "d7908185c1925a527407bfa72a6b7c50"
server_key_iv: "d41bb1ff0f91e7f89e98622feff67ca7"
server_key_version: "encryptionkey@0"
I20260808 15:58:13.582737 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "25ed9653d16a4dcba9808d07e4ac8c99"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "d7908185c1925a527407bfa72a6b7c50"
server_key_iv: "d41bb1ff0f91e7f89e98622feff67ca7"
server_key_version: "encryptionkey@0"
I20260808 15:58:13.587620 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:58:13.590983 10317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:13.592180 10302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:58:13.592339 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "25ed9653d16a4dcba9808d07e4ac8c99"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "d7908185c1925a527407bfa72a6b7c50"
server_key_iv: "d41bb1ff0f91e7f89e98622feff67ca7"
server_key_version: "encryptionkey@0"
I20260808 15:58:13.592535 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:13.604607 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:13.608817 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:13.609102 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:13.610121 10302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:13.611425 10302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:13.611497 10302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:13.611665 10302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:13.611716 10302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:13.645239 10302 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:43291
I20260808 15:58:13.645277 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:43291 every 8 connection(s)
I20260808 15:58:13.646582 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:58:13.648653 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10302
I20260808 15:58:13.648844 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:58:13.649483 28664 external_mini_cluster.cc:1428] Setting key fdbaabafebb870785e2d958d0041567a
Aug 08 15:58:13 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:13.668738 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:13.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:40355 (local address 127.27.254.62:36521)
0808 15:58:13.649514 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:13.649519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:13.651068 (+ 1549us) server_negotiation.cc:407] Connection header received
0808 15:58:13.653031 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:13.653035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:13.653144 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:13.653254 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:13.654757 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.655495 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.657078 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.657904 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.658049 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:13.661948 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:13.661957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:13.661965 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:13.662010 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:13.665621 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:13.666309 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:13.666316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:13.666322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:13.666468 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:13.666854 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:13.666861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:13.666867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:13.667260 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:13.667419 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:13.668047 (+ 628us) server_negotiation.cc:299] Negotiation successful
0808 15:58:13.668310 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:58:13.669445 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:13.650008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:40355)
0808 15:58:13.650895 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:13.650949 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:13.652776 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:13.653527 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:13.653541 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:13.654040 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:13.654590 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:13.654606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.655641 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.655646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:13.656888 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:13.656903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:13.658104 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:13.658109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:13.658307 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:13.659015 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:13.659037 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:13.661722 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:13.665747 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:13.665753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:13.665765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:13.666170 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:13.666569 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:13.666572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:13.666574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:13.666739 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:13.667377 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:13.667382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:13.667671 (+ 289us) client_negotiation.cc:769] Sending connection context
0808 15:58:13.667900 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 15:58:13.668279 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":330,"threads_started":1}
I20260808 15:58:13.671064 10431 heartbeater.cc:347] Connected to a master server at 127.27.254.62:36521
I20260808 15:58:13.671418 10431 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:13.672137 10431 heartbeater.cc:511] Master 127.27.254.62:36521 requested a full tablet report, sending...
I20260808 15:58:13.673743 10105 ts_manager.cc:194] Registered new tserver with Master: 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291)
I20260808 15:58:13.674559 10105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:40355
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:58:13.706488 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:36521
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:14.042356 10436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:14.042618 10436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:14.042713 10436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:14.049630 10436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:14.049729 10436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:14.049875 10436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:58:14.059695 10436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:14.060863 10436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:14.062162 10436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:14.075343 10444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:14.075449 10441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:14.075416 10442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:14.076198 10436 server_base.cc:1034] running on GCE node
I20260808 15:58:14.076685 10436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:14.077534 10436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:14.078713 10436 hybrid_clock.cc:648] HybridClock initialized: now 1786204694078670 us; error 33 us; skew 500 ppm
I20260808 15:58:14.079056 10436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:14 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:14.086244 10436 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:58:14.087604 10436 webserver.cc:460] Webserver started at http://127.27.254.3:42243/ using document root <none> and password file <none>
I20260808 15:58:14.088258 10436 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:14.088392 10436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:14.088702 10436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:14.091630 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
format_stamp: "Formatted at 2026-08-08 15:58:14 on dist-test-slave-srgv"
server_key: "2916454aebfc7838cfc3c24e83823136"
server_key_iv: "17169c7d8d0c348df59565b5eef65398"
server_key_version: "encryptionkey@0"
I20260808 15:58:14.092379 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
format_stamp: "Formatted at 2026-08-08 15:58:14 on dist-test-slave-srgv"
server_key: "2916454aebfc7838cfc3c24e83823136"
server_key_iv: "17169c7d8d0c348df59565b5eef65398"
server_key_version: "encryptionkey@0"
I20260808 15:58:14.097517 10436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:58:14.100704 10451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:14.102159 10436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:14.102372 10436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
format_stamp: "Formatted at 2026-08-08 15:58:14 on dist-test-slave-srgv"
server_key: "2916454aebfc7838cfc3c24e83823136"
server_key_iv: "17169c7d8d0c348df59565b5eef65398"
server_key_version: "encryptionkey@0"
I20260808 15:58:14.102571 10436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:14.128882 10436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:14.132962 10436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:14.133265 10436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:14.134236 10436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:14.135560 10436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:14.135632 10436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:14.135802 10436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:14.135854 10436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:14.168491 10436 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:44089
I20260808 15:58:14.168578 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:44089 every 8 connection(s)
I20260808 15:58:14.169878 10436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:58:14.170948 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10436
I20260808 15:58:14.171168 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:58:14.171607 28664 external_mini_cluster.cc:1428] Setting key 033c6f60c1d65212e5e9e864a9a81b1c
Aug 08 15:58:14 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204694, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:14.184746 10296 heartbeater.cc:503] Master 127.27.254.62:36521 was elected leader, sending a full tablet report...
I20260808 15:58:14.190126 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.172483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:48157 (local address 127.27.254.62:36521)
0808 15:58:14.173163 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.173170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.174201 (+ 1031us) server_negotiation.cc:407] Connection header received
0808 15:58:14.176210 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.176214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.176277 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.176365 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:14.177851 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.178623 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.180102 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.180491 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.180583 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.183979 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:14.183984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:14.183987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:14.184018 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:14.187496 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.188150 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.188153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.188155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.188273 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.188634 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.188637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.188639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.188882 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:14.188993 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.189692 (+ 699us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.189862 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":386,"threads_started":1}
I20260808 15:58:14.191104 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.173140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:48157)
0808 15:58:14.174052 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:14.174103 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:14.175965 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:14.176558 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:14.176565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:14.177054 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:14.177703 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.177722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.178778 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.178786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.179979 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.179994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.180700 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.180706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.180908 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.181604 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:14.181625 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:14.183780 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:14.187617 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:14.187623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:14.187635 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:14.188045 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:14.188358 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:14.188361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:14.188363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:14.188534 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:14.188984 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:14.188990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:14.189303 (+ 313us) client_negotiation.cc:769] Sending connection context
0808 15:58:14.189589 (+ 286us) client_negotiation.cc:240] Negotiation successful
0808 15:58:14.189995 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":434,"threads_started":1}
I20260808 15:58:14.192425 10565 heartbeater.cc:347] Connected to a master server at 127.27.254.62:36521
I20260808 15:58:14.192725 10565 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:14.193485 10565 heartbeater.cc:511] Master 127.27.254.62:36521 requested a full tablet report, sending...
I20260808 15:58:14.194810 10105 ts_manager.cc:194] Registered new tserver with Master: fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
I20260808 15:58:14.195639 10105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:48157
I20260808 15:58:14.199559 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:58:14.216586 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.204188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.27.254.62:36521)
0808 15:58:14.204359 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.204363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.204772 (+ 409us) server_negotiation.cc:407] Connection header received
0808 15:58:14.204935 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.204938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.205007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.205085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:14.206532 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.207454 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.208858 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.209252 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.209352 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.211639 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:14.211644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:14.211647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:14.211676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:14.214327 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.214971 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.214975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.214977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.215098 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.215403 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.215406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.215408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.215721 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:14.215905 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.216182 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.216344 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:58:14.220389 10105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:58:14.227198 10105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:58:14.265945 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.238911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.27.254.1:37009)
0808 15:58:14.255549 (+ 16638us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.255559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.255592 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:58:14.255705 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.255714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.256044 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.257927 (+ 1883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:14.258154 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.259918 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.263157 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.264577 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.264709 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.264963 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.265423 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.265584 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16412,"thread_start_us":17988,"threads_started":1}
I20260808 15:58:14.265983 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.238824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:47906)
0808 15:58:14.249981 (+ 11157us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:14.250040 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:14.250280 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:14.257223 (+ 6943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:14.257230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:14.257260 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:14.257701 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.257717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.260042 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.260049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.262982 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.262998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.264790 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.264796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.265108 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.265175 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 15:58:14.265332 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 15:58:14.265582 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10772,"mutex_wait_us":121,"thread_start_us":17517,"threads_started":1}
I20260808 15:58:14.266263 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.239833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.27.254.3:44089)
0808 15:58:14.257009 (+ 17176us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.257017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.257048 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:58:14.257143 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.257154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.257396 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.257794 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:14.258739 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.260263 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.263084 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.264582 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.264709 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.264960 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.265778 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.265941 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16974,"thread_start_us":17565,"threads_started":1}
I20260808 15:58:14.266791 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.238933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:44089 (local address 127.0.0.1:35158)
0808 15:58:14.256672 (+ 17739us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:14.256694 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:14.256826 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:14.258087 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:14.258092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:14.258118 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:14.258540 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.258552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.260480 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.260485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.262934 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.262946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.265058 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.265063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.265288 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.265321 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:58:14.265459 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 15:58:14.265598 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17563,"mutex_wait_us":101,"thread_start_us":10462,"threads_started":1}
I20260808 15:58:14.270254 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.256800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:43291 (local address 127.0.0.1:47378)
0808 15:58:14.260628 (+ 3828us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:14.260647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:14.260793 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:14.261711 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:14.261716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:14.261743 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:14.262223 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.262235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.263813 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.263820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.265552 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.265564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.267629 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.267635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.269636 (+ 2001us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.269668 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:58:14.269773 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 15:58:14.269904 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3674,"mutex_wait_us":1,"thread_start_us":3471,"threads_started":1}
I20260808 15:58:14.270321 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.250831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.27.254.2:43291)
0808 15:58:14.260014 (+ 9183us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.260021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.260775 (+ 754us) server_negotiation.cc:407] Connection header received
0808 15:58:14.260960 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.260968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.261185 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.261385 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:14.262418 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.263669 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.265719 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.267402 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.267556 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.267736 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.269859 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.270032 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9026,"thread_start_us":8911,"threads_started":1}
I20260808 15:58:14.270180 10231 tablet_service.cc:1467] Processing CreateTablet for tablet bce4c60efd6240e5ae3262dc9bf65a8c (DEFAULT_TABLE table=test-table [id=d07527fbdd294c489a2728ac2a38d4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:14.270318 10499 tablet_service.cc:1467] Processing CreateTablet for tablet bce4c60efd6240e5ae3262dc9bf65a8c (DEFAULT_TABLE table=test-table [id=d07527fbdd294c489a2728ac2a38d4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:14.271729 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce4c60efd6240e5ae3262dc9bf65a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:14.273295 10366 tablet_service.cc:1467] Processing CreateTablet for tablet bce4c60efd6240e5ae3262dc9bf65a8c (DEFAULT_TABLE table=test-table [id=d07527fbdd294c489a2728ac2a38d4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:14.274212 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce4c60efd6240e5ae3262dc9bf65a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:14.274443 10366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce4c60efd6240e5ae3262dc9bf65a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:14.282644 10584 tablet_bootstrap.cc:492] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Bootstrap starting.
I20260808 15:58:14.285748 10584 tablet_bootstrap.cc:654] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:14.286725 10584 log.cc:824] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:14.289505 10587 tablet_bootstrap.cc:492] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Bootstrap starting.
I20260808 15:58:14.289916 10584 tablet_bootstrap.cc:492] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: No bootstrap required, opened a new log
I20260808 15:58:14.290213 10584 ts_tablet_manager.cc:1397] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 15:58:14.290810 10586 tablet_bootstrap.cc:492] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Bootstrap starting.
I20260808 15:58:14.292843 10587 tablet_bootstrap.cc:654] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:14.293851 10587 log.cc:824] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:14.294108 10586 tablet_bootstrap.cc:654] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:14.295287 10586 log.cc:824] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:14.296577 10587 tablet_bootstrap.cc:492] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: No bootstrap required, opened a new log
I20260808 15:58:14.296806 10587 ts_tablet_manager.cc:1397] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 15:58:14.298286 10586 tablet_bootstrap.cc:492] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: No bootstrap required, opened a new log
I20260808 15:58:14.298525 10586 ts_tablet_manager.cc:1397] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:58:14.301062 10584 raft_consensus.cc:348] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.301494 10584 raft_consensus.cc:374] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:14.301594 10584 raft_consensus.cc:729] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ffc267cc4e488ea64b1a1a2c86e1fd, State: Initialized, Role: FOLLOWER
I20260808 15:58:14.302178 10584 consensus_queue.cc:260] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.303977 10584 ts_tablet_manager.cc:1428] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 15:58:14.305334 10587 raft_consensus.cc:348] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.305725 10587 raft_consensus.cc:374] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:14.305828 10587 raft_consensus.cc:729] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5178e9c2ca49db9638ba441d8d94cc, State: Initialized, Role: FOLLOWER
I20260808 15:58:14.306401 10587 consensus_queue.cc:260] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.308065 10587 ts_tablet_manager.cc:1428] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 15:58:14.308468 10565 heartbeater.cc:503] Master 127.27.254.62:36521 was elected leader, sending a full tablet report...
I20260808 15:58:14.309186 10586 raft_consensus.cc:348] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.309697 10586 raft_consensus.cc:374] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:14.309856 10586 raft_consensus.cc:729] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ed9653d16a4dcba9808d07e4ac8c99, State: Initialized, Role: FOLLOWER
I20260808 15:58:14.310330 10586 consensus_queue.cc:260] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.311275 10431 heartbeater.cc:503] Master 127.27.254.62:36521 was elected leader, sending a full tablet report...
I20260808 15:58:14.311887 10586 ts_tablet_manager.cc:1428] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260808 15:58:14.398625 10432 tablet.cc:2369] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:58:14.406268 10297 tablet.cc:2369] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:58:14.421649 10566 tablet.cc:2369] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:14.507228 10590 raft_consensus.cc:482] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:14.507655 10590 raft_consensus.cc:504] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.509577 10590 leader_election.cc:288] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089), 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291)
I20260808 15:58:14.516702 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.509979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:59491 (local address 127.27.254.3:44089)
0808 15:58:14.510174 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.510179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.510780 (+ 601us) server_negotiation.cc:407] Connection header received
0808 15:58:14.511082 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.511090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.511174 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.511288 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:14.512008 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.512963 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.514940 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.515744 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.515862 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.515914 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.516318 (+ 404us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.516447 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:58:14.516747 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.511109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:33985 (local address 127.27.254.2:43291)
0808 15:58:14.511310 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.511316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.511595 (+ 279us) server_negotiation.cc:407] Connection header received
0808 15:58:14.511718 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.511725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.511808 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.511910 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:14.512536 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.513386 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.515105 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.515885 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.516045 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.516112 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.516387 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.516516 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:58:14.516811 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.510832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:43291 (local address 127.27.254.1:33985)
0808 15:58:14.511486 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:14.511498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:14.511610 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:14.511957 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:14.511962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:14.511988 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:14.512363 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.512398 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.513530 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.513534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.514958 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.514971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.516092 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.516097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.516210 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.516255 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 15:58:14.516346 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 15:58:14.516457 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":363,"threads_started":1}
I20260808 15:58:14.517033 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.509980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:44089 (local address 127.27.254.1:59491)
0808 15:58:14.510667 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:14.510688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:14.510846 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:14.511406 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:14.511413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:14.511456 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:14.511844 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.511859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.513088 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.513093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.514771 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:14.514783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.515948 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.515956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:14.516145 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.516182 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:58:14.516254 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 15:58:14.516369 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":45,"thread_start_us":389,"threads_started":1}
I20260808 15:58:14.517737 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4c60efd6240e5ae3262dc9bf65a8c" candidate_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" is_pre_election: true
I20260808 15:58:14.517930 10520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4c60efd6240e5ae3262dc9bf65a8c" candidate_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" is_pre_election: true
I20260808 15:58:14.518239 10386 raft_consensus.cc:2454] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ffc267cc4e488ea64b1a1a2c86e1fd in term 0.
I20260808 15:58:14.518426 10520 raft_consensus.cc:2454] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ffc267cc4e488ea64b1a1a2c86e1fd in term 0.
I20260808 15:58:14.518914 10184 leader_election.cc:302] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd, 25ed9653d16a4dcba9808d07e4ac8c99; no voters:
I20260808 15:58:14.519249 10590 raft_consensus.cc:2781] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:58:14.519385 10590 raft_consensus.cc:482] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:58:14.519495 10590 raft_consensus.cc:3037] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:14.521687 10590 raft_consensus.cc:504] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.522224 10590 leader_election.cc:288] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 1 election: Requested vote from peers fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089), 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291)
I20260808 15:58:14.522607 10520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4c60efd6240e5ae3262dc9bf65a8c" candidate_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
I20260808 15:58:14.522667 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4c60efd6240e5ae3262dc9bf65a8c" candidate_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ed9653d16a4dcba9808d07e4ac8c99"
I20260808 15:58:14.522786 10520 raft_consensus.cc:3037] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:14.522863 10386 raft_consensus.cc:3037] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:14.524959 10386 raft_consensus.cc:2454] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ffc267cc4e488ea64b1a1a2c86e1fd in term 1.
I20260808 15:58:14.524976 10520 raft_consensus.cc:2454] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ffc267cc4e488ea64b1a1a2c86e1fd in term 1.
I20260808 15:58:14.525343 10184 leader_election.cc:302] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd, fd5178e9c2ca49db9638ba441d8d94cc; no voters:
I20260808 15:58:14.525662 10590 raft_consensus.cc:2781] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:14.526114 10590 raft_consensus.cc:686] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 LEADER]: Becoming Leader. State: Replica: 19ffc267cc4e488ea64b1a1a2c86e1fd, State: Running, Role: LEADER
I20260808 15:58:14.526564 10590 consensus_queue.cc:237] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } }
I20260808 15:58:14.531473 10105 catalog_manager.cc:5495] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1). New cstate: current_term: 1 leader_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } health_report { overall_health: UNKNOWN } } }
I20260808 15:58:14.614961 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.609554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.27.254.1:37009)
0808 15:58:14.609768 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.609773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.609790 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:58:14.609876 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.609879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.609977 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.610099 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:58:14.610457 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.611161 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.613076 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.613509 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.613601 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.614026 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:58:14.614528 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:58:14.614631 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.614701 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.614761 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:58:14.624401 10386 raft_consensus.cc:1261] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Refusing update from remote peer 19ffc267cc4e488ea64b1a1a2c86e1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:14.624501 10520 raft_consensus.cc:1261] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Refusing update from remote peer 19ffc267cc4e488ea64b1a1a2c86e1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:14.625593 10595 consensus_queue.cc:1035] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:14.626132 10590 consensus_queue.cc:1035] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:14.649639 10600 mvcc.cc:205] Tried to move back new op lower bound from 7316294429169369088 to 7316294428783984640. Current Snapshot: MvccSnapshot[applied={T|T < 7316294429169369088}]
I20260808 15:58:14.663049 10105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:14.663355 10105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:14.666742 10105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Sending DeleteTablet for 3 replicas of tablet bce4c60efd6240e5ae3262dc9bf65a8c
I20260808 15:58:14.667771 10231 tablet_service.cc:1514] Processing DeleteTablet for tablet bce4c60efd6240e5ae3262dc9bf65a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:14 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:47906
I20260808 15:58:14.668087 10366 tablet_service.cc:1514] Processing DeleteTablet for tablet bce4c60efd6240e5ae3262dc9bf65a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:14 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:47378
I20260808 15:58:14.668175 10499 tablet_service.cc:1514] Processing DeleteTablet for tablet bce4c60efd6240e5ae3262dc9bf65a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:14 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:35158
I20260808 15:58:14.668586 10608 tablet_replica.cc:330] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: stopping tablet replica
I20260808 15:58:14.668954 10608 raft_consensus.cc:2229] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:58:14.669080 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10167
I20260808 15:58:14.669584 10608 raft_consensus.cc:2258] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:14.677733 10608 ts_tablet_manager.cc:1905] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:58:14 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:58:14.686865 10610 tablet_replica.cc:330] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: stopping tablet replica
I20260808 15:58:14.687439 10609 tablet_replica.cc:330] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: stopping tablet replica
I20260808 15:58:14.688001 10609 raft_consensus.cc:2229] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:14.688519 10610 raft_consensus.cc:2229] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:14.689144 10609 raft_consensus.cc:2258] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:14.689680 10610 raft_consensus.cc:2258] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:14.692157 10610 ts_tablet_manager.cc:1905] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:14.692483 10608 ts_tablet_manager.cc:1918] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:14.692653 10608 log.cc:1196] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/bce4c60efd6240e5ae3262dc9bf65a8c
I20260808 15:58:14.693222 10608 ts_tablet_manager.cc:1939] T bce4c60efd6240e5ae3262dc9bf65a8c P 19ffc267cc4e488ea64b1a1a2c86e1fd: Deleting consensus metadata
I20260808 15:58:14.693723 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.670255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.27.254.1:37009)
0808 15:58:14.670413 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.670419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.671058 (+ 639us) server_negotiation.cc:407] Connection header received
0808 15:58:14.671267 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.671274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.671350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.671434 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:14.673650 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.674686 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.676232 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.676912 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.677043 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.683401 (+ 6358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:14.683421 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:14.683442 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:14.683511 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:14.688448 (+ 4937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.689554 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.689562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.689565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.689793 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.690764 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.690770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.690773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.691279 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:14.691571 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.693027 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.693280 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:58:14.695297 10092 catalog_manager.cc:4841] TS 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1:37009): tablet bce4c60efd6240e5ae3262dc9bf65a8c (table test-table [id=d07527fbdd294c489a2728ac2a38d4e3]) successfully deleted
I20260808 15:58:14.695399 10609 ts_tablet_manager.cc:1905] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:14.695880 10291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:14.700260 10610 ts_tablet_manager.cc:1918] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:14.700562 10610 log.cc:1196] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/bce4c60efd6240e5ae3262dc9bf65a8c
I20260808 15:58:14.701130 10610 ts_tablet_manager.cc:1939] T bce4c60efd6240e5ae3262dc9bf65a8c P fd5178e9c2ca49db9638ba441d8d94cc: Deleting consensus metadata
I20260808 15:58:14.703002 10609 ts_tablet_manager.cc:1918] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:14.703173 10609 log.cc:1196] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/bce4c60efd6240e5ae3262dc9bf65a8c
I20260808 15:58:14.703106 10090 catalog_manager.cc:4841] TS fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): tablet bce4c60efd6240e5ae3262dc9bf65a8c (table test-table [id=d07527fbdd294c489a2728ac2a38d4e3]) successfully deleted
I20260808 15:58:14.703673 10609 ts_tablet_manager.cc:1939] T bce4c60efd6240e5ae3262dc9bf65a8c P 25ed9653d16a4dcba9808d07e4ac8c99: Deleting consensus metadata
I20260808 15:58:14.705036 10090 catalog_manager.cc:4841] TS 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): tablet bce4c60efd6240e5ae3262dc9bf65a8c (table test-table [id=d07527fbdd294c489a2728ac2a38d4e3]) successfully deleted
I20260808 15:58:14.793269 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10167
I20260808 15:58:14.812201 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10302
Aug 08 15:58:14 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:58:14.826874 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.813003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.27.254.2:43291)
0808 15:58:14.813174 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.813180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.813198 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:58:14.813329 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.813333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.813393 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.813540 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:14.814775 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.815445 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.816667 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.817071 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.817158 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.821277 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:14.821290 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:14.821305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:14.821334 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:14.824003 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.824735 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.824743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.824745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.824874 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.825264 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.825270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.825273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.825703 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:14.826016 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.826298 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.826513 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:58:14.827625 10426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:14.916641 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10302
I20260808 15:58:14.933589 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10436
Aug 08 15:58:14 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:58:14.948249 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:14.934349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.27.254.3:44089)
0808 15:58:14.934523 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:14.934528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:14.934547 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:58:14.934681 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:14.934686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:14.934750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:14.934836 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:14.936118 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.936828 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.938197 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:14.938574 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:14.938668 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:14.942851 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:14.942866 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:14.942882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:14.942918 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:14.945662 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.946331 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.946338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.946341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.946489 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:14.946822 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:14.946826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:14.946827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:14.947192 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:14.947424 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:14.947688 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:58:14.947870 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:58:14.948952 10560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:15.038053 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10436
I20260808 15:58:15.056314 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10072
I20260808 15:58:15.057497 10135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:15.169360 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10072
I20260808 15:58:15.190649 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36521
--webserver_interface=127.27.254.62
--webserver_port=36819
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:15.518568 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:15.518867 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:15.518996 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:15.526288 10616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:15.526408 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:15.526454 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:15.526505 10616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:15.526595 10616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:15.536584 10616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=36819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:15.538038 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:15.539438 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:15.564188 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:15.564226 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:15.564203 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:15.565156 10616 server_base.cc:1034] running on GCE node
I20260808 15:58:15.566143 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:15.567777 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:15.569024 10616 hybrid_clock.cc:648] HybridClock initialized: now 1786204695568967 us; error 65 us; skew 500 ppm
I20260808 15:58:15.569505 10616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:15 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204695, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:15.573071 10616 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:15.574867 10616 webserver.cc:460] Webserver started at http://127.27.254.62:36819/ using document root <none> and password file <none>
I20260808 15:58:15.575630 10616 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:15.575799 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:15.581970 10616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:58:15.585160 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:15.586621 10616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:15.586826 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "9c2f3b6d642549a1b7e84c8d1abf89da"
format_stamp: "Formatted at 2026-08-08 15:58:12 on dist-test-slave-srgv"
server_key: "54225eba4446fca2d18e594ea2e07034"
server_key_iv: "824ae82b08600bc55ab6c1783f972e76"
server_key_version: "encryptionkey@0"
I20260808 15:58:15.587916 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:15.600199 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:15.604602 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:15.604894 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:15.623898 10616 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:36521
I20260808 15:58:15.623936 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:36521 every 8 connection(s)
I20260808 15:58:15.625478 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:15.632493 10685 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:58:15.635334 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10616
I20260808 15:58:15.636369 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap starting.
I20260808 15:58:15.636372 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:37009
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:36521
--webserver_port=36201
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:58:15.658695 10685 log.cc:824] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:15.670763 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:58:15.671329 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap complete.
I20260808 15:58:15.680781 10685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:15.681687 10685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2f3b6d642549a1b7e84c8d1abf89da, State: Initialized, Role: FOLLOWER
I20260808 15:58:15.682310 10685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:15.682548 10685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:58:15.682662 10685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:58:15.682883 10685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:58:15.686754 10685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:15.687279 10685 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2f3b6d642549a1b7e84c8d1abf89da; no voters:
I20260808 15:58:15.687915 10685 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:58:15.688056 10689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:58:15.689518 10689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 LEADER]: Becoming Leader. State: Replica: 9c2f3b6d642549a1b7e84c8d1abf89da, State: Running, Role: LEADER
I20260808 15:58:15.690047 10689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:15.690400 10685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:15.697358 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } } }
I20260808 15:58:15.697860 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:15.698436 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2f3b6d642549a1b7e84c8d1abf89da. Latest consensus state: current_term: 2 leader_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } } }
I20260808 15:58:15.698750 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:15.706841 10697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:15.715070 10697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:15.716673 10697 catalog_manager.cc:1259] Loaded cluster ID: cd43036ac2a44d01994f8453636005d9
I20260808 15:58:15.716791 10697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:58:15.719949 10697 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 15:58:15.737556 10697 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 15:58:15.753389 10697 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:58:15.755174 10697 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Loaded TSK: 0
I20260808 15:58:15.756693 10697 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:58:16.002665 10687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:16.003002 10687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:16.003170 10687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:16.010452 10687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:16.010600 10687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:16.010776 10687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:58:16.020682 10687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:37009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=36201
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:16.022048 10687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:16.023414 10687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:16.038524 10713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:16.038501 10714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:16.039590 10687 server_base.cc:1034] running on GCE node
W20260808 15:58:16.039772 10716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:16.040479 10687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:16.041546 10687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:16.042789 10687 hybrid_clock.cc:648] HybridClock initialized: now 1786204696042724 us; error 57 us; skew 500 ppm
I20260808 15:58:16.043107 10687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:16 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:16.046432 10687 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:58:16.048013 10687 webserver.cc:460] Webserver started at http://127.27.254.1:36201/ using document root <none> and password file <none>
I20260808 15:58:16.048694 10687 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:16.048857 10687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:16.054358 10687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:58:16.057159 10723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.058655 10687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:58:16.058825 10687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "19db8403f67d4adf35834ba08472f404"
server_key_iv: "48c5cad7cc593a091b365b570f27ff56"
server_key_version: "encryptionkey@0"
I20260808 15:58:16.059865 10687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:16.076277 10687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:16.080672 10687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:16.080948 10687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:16.082048 10687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:16.083470 10687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:16.083556 10687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.083709 10687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:16.083814 10687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.117132 10687 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:37009
I20260808 15:58:16.117188 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:37009 every 8 connection(s)
I20260808 15:58:16.118593 10687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:58:16.124264 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10687
I20260808 15:58:16.124912 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:43291
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:36521
--webserver_port=33415
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:58:16 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:16.142645 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:16.121491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:46901 (local address 127.27.254.62:36521)
0808 15:58:16.122038 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:16.122050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:16.122729 (+ 679us) server_negotiation.cc:407] Connection header received
0808 15:58:16.124869 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:16.124881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:16.125268 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:16.125675 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:16.127180 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.128382 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.130034 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.130673 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.130913 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:16.134874 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:16.134882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:16.134895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:16.134928 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:16.138034 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:16.139034 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:16.139040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:16.139046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:16.139200 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:16.139692 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:16.139696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:16.139697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:16.140167 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:16.140314 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:16.141313 (+ 999us) server_negotiation.cc:299] Negotiation successful
0808 15:58:16.141707 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260808 15:58:16.143083 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:16.121801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:46901)
0808 15:58:16.122559 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:16.122596 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:16.124311 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:16.125807 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:16.125820 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:16.126416 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:16.126947 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:16.126965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.128547 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.128553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:16.129885 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:16.129898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.130944 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.130952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:16.131221 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:16.131913 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:16.131935 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:16.134603 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:16.138203 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:16.138231 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:16.138249 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:16.138894 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:16.139317 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:16.139324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:16.139330 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:16.139572 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:16.140308 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:16.140318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:16.140724 (+ 406us) client_negotiation.cc:769] Sending connection context
0808 15:58:16.141053 (+ 329us) client_negotiation.cc:240] Negotiation successful
0808 15:58:16.141704 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":317,"threads_started":1}
I20260808 15:58:16.145905 10836 heartbeater.cc:347] Connected to a master server at 127.27.254.62:36521
I20260808 15:58:16.146207 10836 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:16.146939 10836 heartbeater.cc:511] Master 127.27.254.62:36521 requested a full tablet report, sending...
I20260808 15:58:16.149041 10650 ts_manager.cc:194] Registered new tserver with Master: 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1:37009)
I20260808 15:58:16.150624 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:46901
W20260808 15:58:16.454592 10840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:16.454864 10840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:16.455019 10840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:16.462231 10840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:16.462363 10840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:16.462546 10840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:58:16.472355 10840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:43291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=33415
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:16.473693 10840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:16.474990 10840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:16.488157 10846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:16.488125 10849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:16.488225 10847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:16.488958 10840 server_base.cc:1034] running on GCE node
I20260808 15:58:16.489662 10840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:16.490685 10840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:16.491894 10840 hybrid_clock.cc:648] HybridClock initialized: now 1786204696491832 us; error 55 us; skew 500 ppm
I20260808 15:58:16.492209 10840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:16 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:16.495400 10840 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:58:16.496835 10840 webserver.cc:460] Webserver started at http://127.27.254.2:33415/ using document root <none> and password file <none>
I20260808 15:58:16.497593 10840 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:16.497751 10840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:16.503181 10840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:58:16.505885 10856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.507242 10840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:58:16.507522 10840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "25ed9653d16a4dcba9808d07e4ac8c99"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "d7908185c1925a527407bfa72a6b7c50"
server_key_iv: "d41bb1ff0f91e7f89e98622feff67ca7"
server_key_version: "encryptionkey@0"
I20260808 15:58:16.508494 10840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:16.524741 10840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:16.528505 10840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:16.528784 10840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:16.529758 10840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:16.531036 10840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:16.531153 10840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.531286 10840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:16.531373 10840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.566864 10840 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:43291
I20260808 15:58:16.566951 10970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:43291 every 8 connection(s)
I20260808 15:58:16.568383 10840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:58:16.571281 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10840
I20260808 15:58:16.572083 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:44089
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:36521
--webserver_port=42243
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:58:16 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:16.589882 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:16.570949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:51355 (local address 127.27.254.62:36521)
0808 15:58:16.571135 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:16.571140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:16.572603 (+ 1463us) server_negotiation.cc:407] Connection header received
0808 15:58:16.574630 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:16.574635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:16.574730 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:16.574844 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:16.576395 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.577102 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.578716 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.579132 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.579268 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:16.583718 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:16.583723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:16.583726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:16.583759 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:16.586356 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:16.587367 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:16.587371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:16.587373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:16.587507 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:16.588025 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:16.588029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:16.588030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:16.588288 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:16.588402 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:16.589358 (+ 956us) server_negotiation.cc:299] Negotiation successful
0808 15:58:16.589609 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:58:16.591660 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:16.571631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:51355)
0808 15:58:16.572429 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:16.572467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:16.574353 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:16.575099 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:16.575108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:16.575619 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:16.576201 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:16.576218 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.577284 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.577288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:16.578555 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:16.578566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:16.579351 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:16.579359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:16.579646 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:16.580405 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:16.580438 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:16.583483 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:16.586498 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:16.586510 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:16.586529 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:16.587229 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:16.587618 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:16.587626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:16.587632 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:16.587905 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:16.588447 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:16.588459 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:16.588951 (+ 492us) client_negotiation.cc:769] Sending connection context
0808 15:58:16.589272 (+ 321us) client_negotiation.cc:240] Negotiation successful
0808 15:58:16.589930 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260808 15:58:16.593454 10971 heartbeater.cc:347] Connected to a master server at 127.27.254.62:36521
I20260808 15:58:16.593787 10971 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:16.594552 10971 heartbeater.cc:511] Master 127.27.254.62:36521 requested a full tablet report, sending...
I20260808 15:58:16.596045 10650 ts_manager.cc:194] Registered new tserver with Master: 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291)
I20260808 15:58:16.596856 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:51355
W20260808 15:58:16.913362 10974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:16.913731 10974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:16.913902 10974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:16.921017 10974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:16.921159 10974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:16.921335 10974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:58:16.930701 10974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:44089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=42243
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:16.931972 10974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:16.933336 10974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:16.946652 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:16.946686 10980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:16.946627 10982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:16.947544 10974 server_base.cc:1034] running on GCE node
I20260808 15:58:16.948133 10974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:16.949021 10974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:16.950214 10974 hybrid_clock.cc:648] HybridClock initialized: now 1786204696950163 us; error 38 us; skew 500 ppm
I20260808 15:58:16.950542 10974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:16 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:16.953718 10974 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:58:16.955279 10974 webserver.cc:460] Webserver started at http://127.27.254.3:42243/ using document root <none> and password file <none>
I20260808 15:58:16.955979 10974 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:16.956137 10974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:16.961298 10974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:16.964010 10989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.965556 10974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:16.965725 10974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
format_stamp: "Formatted at 2026-08-08 15:58:14 on dist-test-slave-srgv"
server_key: "2916454aebfc7838cfc3c24e83823136"
server_key_iv: "17169c7d8d0c348df59565b5eef65398"
server_key_version: "encryptionkey@0"
I20260808 15:58:16.966681 10974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:16.980754 10974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:16.984513 10974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:16.984807 10974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:16.985812 10974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:16.987203 10974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:16.987337 10974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:16.987478 10974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:16.987567 10974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:17.021111 10974 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:44089
I20260808 15:58:17.021165 11102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:44089 every 8 connection(s)
I20260808 15:58:17.022518 10974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:58:17.027786 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 10974
Aug 08 15:58:17 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:17.044039 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.028706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.27.254.62:36521)
0808 15:58:17.029181 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.029187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.029530 (+ 343us) server_negotiation.cc:407] Connection header received
0808 15:58:17.029718 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.029723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.029811 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.029932 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:17.031870 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.032562 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.034385 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.034978 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.035117 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.038246 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:17.038250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:17.038253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:17.038290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:17.041027 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.041799 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.041803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.041805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.041973 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.042471 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.042476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.042479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.042790 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:17.043060 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.043376 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.043612 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":362,"spinlock_wait_cycles":1280,"thread_start_us":247,"threads_started":1}
I20260808 15:58:17.044817 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.025073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:51427 (local address 127.27.254.62:36521)
0808 15:58:17.025283 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.025288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.026476 (+ 1188us) server_negotiation.cc:407] Connection header received
0808 15:58:17.028251 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.028257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.028344 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.028444 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:17.030255 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.031446 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.033108 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.033778 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.033904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.037547 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:17.037556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:17.037562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:17.037601 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:17.040939 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.041760 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.041766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.041770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.041917 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.042522 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.042525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.042527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.042802 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:17.042990 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.043888 (+ 898us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.044047 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":70}
I20260808 15:58:17.045696 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.025599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:51427)
0808 15:58:17.026315 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.026364 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:17.027995 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:17.028728 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:17.028739 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:17.029520 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:17.030094 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.030111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.031604 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.031607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.032976 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.032987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.033986 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.033991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.034174 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.034824 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:17.034844 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:17.037303 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:17.041078 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:17.041088 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:17.041108 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:17.041632 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:17.042166 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:17.042172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:17.042179 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:17.042374 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:17.042977 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:17.042985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:17.043419 (+ 434us) client_negotiation.cc:769] Sending connection context
0808 15:58:17.043773 (+ 354us) client_negotiation.cc:240] Negotiation successful
0808 15:58:17.044337 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":276,"threads_started":1}
I20260808 15:58:17.047112 11103 heartbeater.cc:347] Connected to a master server at 127.27.254.62:36521
I20260808 15:58:17.047422 11103 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:17.048161 11103 heartbeater.cc:511] Master 127.27.254.62:36521 requested a full tablet report, sending...
I20260808 15:58:17.049576 10649 ts_manager.cc:194] Registered new tserver with Master: fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
I20260808 15:58:17.050472 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:51427
I20260808 15:58:17.058894 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:58:17.065727 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.059817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.27.254.62:36521)
0808 15:58:17.060056 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.060065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.060446 (+ 381us) server_negotiation.cc:407] Connection header received
0808 15:58:17.060619 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.060623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.060745 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.060861 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:58:17.061231 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.062035 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.063611 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.064007 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.064098 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.064632 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:58:17.065090 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:58:17.065199 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.065336 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.065397 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:58:17.066568 10649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:58:17.073902 10649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:58:17.096724 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.085509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.27.254.1:37009)
0808 15:58:17.087780 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.087786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.087808 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:58:17.087882 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.087887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.088080 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.088276 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:17.089300 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.090871 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.093975 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.095449 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.095576 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.095814 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.096185 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.096358 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2123,"thread_start_us":1996,"threads_started":1}
I20260808 15:58:17.096778 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.085302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40666)
0808 15:58:17.086914 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.086962 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:17.087181 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:17.088294 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:17.088301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:17.088331 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:17.088862 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.088878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.091087 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.091095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.093804 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.093819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.095689 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.095695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.095924 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.095962 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:58:17.096134 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 15:58:17.096370 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":1145,"threads_started":1}
I20260808 15:58:17.099740 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.085993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:43291 (local address 127.0.0.1:59644)
0808 15:58:17.088995 (+ 3002us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.089025 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:17.089127 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:17.092137 (+ 3010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:17.092144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:17.092171 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:17.093440 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.093455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.094880 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.094887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.096855 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.096872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.098969 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.098973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.099093 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.099116 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:58:17.099230 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:58:17.099356 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2844,"mutex_wait_us":47,"thread_start_us":2693,"threads_started":1}
I20260808 15:58:17.099764 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.085913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.27.254.2:43291)
0808 15:58:17.091538 (+ 5625us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.091548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.091583 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:58:17.091683 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.091691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.091941 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.092142 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:17.093645 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.094732 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.097063 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.098567 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.098697 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.098833 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.099283 (+ 450us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.099448 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5441,"thread_start_us":5298,"threads_started":1}
I20260808 15:58:17.101258 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.089921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.27.254.3:44089)
0808 15:58:17.090720 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.090728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.092049 (+ 1321us) server_negotiation.cc:407] Connection header received
0808 15:58:17.092296 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.092307 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.092542 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.092739 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:17.093763 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.095392 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.098670 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.100154 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.100284 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.100489 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.100750 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.100916 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":449,"threads_started":1}
I20260808 15:58:17.101143 10771 tablet_service.cc:1467] Processing CreateTablet for tablet 17cae65bb34b4dfbaa9ac5288f91be33 (DEFAULT_TABLE table=test-table [id=0c0f6f0beb8741ee957f9edfbba3dd66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:17.102581 10771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17cae65bb34b4dfbaa9ac5288f91be33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:17.103470 10905 tablet_service.cc:1467] Processing CreateTablet for tablet 17cae65bb34b4dfbaa9ac5288f91be33 (DEFAULT_TABLE table=test-table [id=0c0f6f0beb8741ee957f9edfbba3dd66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:17.104497 10905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17cae65bb34b4dfbaa9ac5288f91be33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:17.105712 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.089203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:44089 (local address 127.0.0.1:33336)
0808 15:58:17.091929 (+ 2726us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.091943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:17.092093 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:17.092805 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:17.092810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:17.093042 (+ 232us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:17.093560 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.093573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.095550 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.095556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.098492 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.098507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.100362 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.100368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.100541 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.100574 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:58:17.100659 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 15:58:17.100775 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2594,"mutex_wait_us":2,"spinlock_wait_cycles":418176,"thread_start_us":2434,"threads_started":1}
I20260808 15:58:17.109190 11038 tablet_service.cc:1467] Processing CreateTablet for tablet 17cae65bb34b4dfbaa9ac5288f91be33 (DEFAULT_TABLE table=test-table [id=0c0f6f0beb8741ee957f9edfbba3dd66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:17.110584 11038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17cae65bb34b4dfbaa9ac5288f91be33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:17.121671 11119 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Bootstrap starting.
I20260808 15:58:17.122956 11120 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Bootstrap starting.
I20260808 15:58:17.126690 11120 tablet_bootstrap.cc:654] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:17.127324 11119 tablet_bootstrap.cc:654] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:17.128039 11120 log.cc:824] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:17.128607 11119 log.cc:824] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:17.134333 11123 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Bootstrap starting.
I20260808 15:58:17.135315 11120 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: No bootstrap required, opened a new log
I20260808 15:58:17.135627 11120 ts_tablet_manager.cc:1397] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260808 15:58:17.135900 11119 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: No bootstrap required, opened a new log
I20260808 15:58:17.136185 11119 ts_tablet_manager.cc:1397] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260808 15:58:17.137538 11123 tablet_bootstrap.cc:654] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:17.138475 11123 log.cc:824] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:17.141145 11123 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: No bootstrap required, opened a new log
I20260808 15:58:17.141400 11123 ts_tablet_manager.cc:1397] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 15:58:17.146548 11120 raft_consensus.cc:348] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.146653 11119 raft_consensus.cc:348] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.147027 11120 raft_consensus.cc:374] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:17.147027 11119 raft_consensus.cc:374] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:17.147126 11119 raft_consensus.cc:729] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ffc267cc4e488ea64b1a1a2c86e1fd, State: Initialized, Role: FOLLOWER
I20260808 15:58:17.147179 11120 raft_consensus.cc:729] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ed9653d16a4dcba9808d07e4ac8c99, State: Initialized, Role: FOLLOWER
I20260808 15:58:17.147816 11119 consensus_queue.cc:260] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.147817 11120 consensus_queue.cc:260] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.148399 11123 raft_consensus.cc:348] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.148814 11123 raft_consensus.cc:374] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:17.148929 11123 raft_consensus.cc:729] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5178e9c2ca49db9638ba441d8d94cc, State: Initialized, Role: FOLLOWER
I20260808 15:58:17.149621 11123 consensus_queue.cc:260] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.150830 10971 heartbeater.cc:503] Master 127.27.254.62:36521 was elected leader, sending a full tablet report...
I20260808 15:58:17.151036 11120 ts_tablet_manager.cc:1428] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20260808 15:58:17.151232 11119 ts_tablet_manager.cc:1428] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260808 15:58:17.151734 10836 heartbeater.cc:503] Master 127.27.254.62:36521 was elected leader, sending a full tablet report...
I20260808 15:58:17.153568 11123 ts_tablet_manager.cc:1428] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260808 15:58:17.153957 11103 heartbeater.cc:503] Master 127.27.254.62:36521 was elected leader, sending a full tablet report...
W20260808 15:58:17.274150 11104 tablet.cc:2369] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:17.311249 11125 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:17.311630 11125 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.313234 11125 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1:37009), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
I20260808 15:58:17.319873 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.313841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:37309 (local address 127.27.254.1:37009)
0808 15:58:17.314036 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.314043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.314572 (+ 529us) server_negotiation.cc:407] Connection header received
0808 15:58:17.314828 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.314839 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.314899 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.315007 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:17.315612 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.316297 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.318193 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.318947 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.319131 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.319210 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.319489 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.319614 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:58:17.320075 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.313546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.27.254.2:37309)
0808 15:58:17.314447 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.314465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:17.314657 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:17.315024 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:17.315029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:17.315107 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:17.315463 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.315473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.316428 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.316434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.318016 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.318027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.319166 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.319173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.319320 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.319342 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:58:17.319468 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 15:58:17.319600 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"spinlock_wait_cycles":24960,"thread_start_us":645,"threads_started":1}
I20260808 15:58:17.320199 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.313802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:42989 (local address 127.27.254.3:44089)
0808 15:58:17.314015 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.314021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.315007 (+ 986us) server_negotiation.cc:407] Connection header received
0808 15:58:17.315206 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.315211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.315268 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.315388 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:17.315970 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.316718 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.318496 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.319264 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.319454 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.319524 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.319813 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.319935 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 15:58:17.320359 10972 tablet.cc:2369] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:17.320516 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.313856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:44089 (local address 127.27.254.2:42989)
0808 15:58:17.314898 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.314912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:17.315062 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:17.315409 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:17.315412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:17.315436 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:17.315822 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.315831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.316864 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.316869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.318346 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:17.318357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.319479 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.319485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:17.319653 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.319676 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:58:17.319786 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 15:58:17.319909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":1,"thread_start_us":709,"threads_started":1}
I20260808 15:58:17.321543 10791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cae65bb34b4dfbaa9ac5288f91be33" candidate_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" is_pre_election: true
I20260808 15:58:17.321583 11058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cae65bb34b4dfbaa9ac5288f91be33" candidate_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" is_pre_election: true
I20260808 15:58:17.322077 10791 raft_consensus.cc:2454] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ed9653d16a4dcba9808d07e4ac8c99 in term 0.
I20260808 15:58:17.322181 11058 raft_consensus.cc:2454] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ed9653d16a4dcba9808d07e4ac8c99 in term 0.
I20260808 15:58:17.322862 10860 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd, 25ed9653d16a4dcba9808d07e4ac8c99, fd5178e9c2ca49db9638ba441d8d94cc; no voters:
I20260808 15:58:17.323276 11125 raft_consensus.cc:2781] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:58:17.323452 11125 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:58:17.323568 11125 raft_consensus.cc:3037] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:17.325723 11125 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.326319 11125 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [CANDIDATE]: Term 1 election: Requested vote from peers 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1:37009), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
I20260808 15:58:17.326721 10791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cae65bb34b4dfbaa9ac5288f91be33" candidate_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd"
I20260808 15:58:17.326850 11058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17cae65bb34b4dfbaa9ac5288f91be33" candidate_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
I20260808 15:58:17.326938 10791 raft_consensus.cc:3037] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:17.327068 11058 raft_consensus.cc:3037] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:17.329082 10791 raft_consensus.cc:2454] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ed9653d16a4dcba9808d07e4ac8c99 in term 1.
I20260808 15:58:17.329176 11058 raft_consensus.cc:2454] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ed9653d16a4dcba9808d07e4ac8c99 in term 1.
I20260808 15:58:17.329514 10860 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd, 25ed9653d16a4dcba9808d07e4ac8c99; no voters:
I20260808 15:58:17.329785 11125 raft_consensus.cc:2781] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:17.330225 11125 raft_consensus.cc:686] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 LEADER]: Becoming Leader. State: Replica: 25ed9653d16a4dcba9808d07e4ac8c99, State: Running, Role: LEADER
I20260808 15:58:17.330660 11125 consensus_queue.cc:237] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:17.334810 10648 catalog_manager.cc:5495] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } health_report { overall_health: UNKNOWN } } }
W20260808 15:58:17.371212 10837 tablet.cc:2369] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:17.391494 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.386192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.27.254.2:43291)
0808 15:58:17.386418 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.386423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.386441 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:58:17.386633 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.386637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.386737 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.386869 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:58:17.387189 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.387874 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.389237 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.389687 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.389832 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.390242 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:58:17.390843 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:58:17.390954 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.391073 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.391173 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:58:17.399693 11058 raft_consensus.cc:1261] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Refusing update from remote peer 25ed9653d16a4dcba9808d07e4ac8c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:17.399693 10791 raft_consensus.cc:1261] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Refusing update from remote peer 25ed9653d16a4dcba9808d07e4ac8c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:17.400688 11130 consensus_queue.cc:1035] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:17.401063 11125 consensus_queue.cc:1035] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:17.414050 11135 mvcc.cc:205] Tried to move back new op lower bound from 7316294440538517504 to 7316294440269791232. Current Snapshot: MvccSnapshot[applied={T|T < 7316294440538517504}]
I20260808 15:58:17.427656 10650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:17.427942 10650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:17.431305 10650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Sending DeleteTablet for 3 replicas of tablet 17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:17.432523 10905 tablet_service.cc:1514] Processing DeleteTablet for tablet 17cae65bb34b4dfbaa9ac5288f91be33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:17 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:59644
I20260808 15:58:17.432503 10771 tablet_service.cc:1514] Processing DeleteTablet for tablet 17cae65bb34b4dfbaa9ac5288f91be33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:17 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:40666
I20260808 15:58:17.433578 11143 tablet_replica.cc:330] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: stopping tablet replica
I20260808 15:58:17.433274 11038 tablet_service.cc:1514] Processing DeleteTablet for tablet 17cae65bb34b4dfbaa9ac5288f91be33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:17 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:33336
I20260808 15:58:17.434082 11143 raft_consensus.cc:2229] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:58:17.434188 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10687
I20260808 15:58:17.434808 11143 raft_consensus.cc:2258] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:17.436909 11143 ts_tablet_manager.cc:1905] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:17.444628 11143 ts_tablet_manager.cc:1918] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:17.444845 11143 log.cc:1196] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:17.445569 11143 ts_tablet_manager.cc:1939] T 17cae65bb34b4dfbaa9ac5288f91be33 P 25ed9653d16a4dcba9808d07e4ac8c99: Deleting consensus metadata
I20260808 15:58:17.447561 10634 catalog_manager.cc:4841] TS 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): tablet 17cae65bb34b4dfbaa9ac5288f91be33 (table test-table [id=0c0f6f0beb8741ee957f9edfbba3dd66]) successfully deleted
I20260808 15:58:17.452301 11145 tablet_replica.cc:330] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: stopping tablet replica
I20260808 15:58:17.453945 11145 raft_consensus.cc:2229] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:17.454032 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.435808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.27.254.1:37009)
0808 15:58:17.435993 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.435998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.436019 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:58:17.436079 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.436085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.436166 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.436264 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:17.438807 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.439995 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.442078 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.442747 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.442869 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.447026 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:17.447042 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:17.447059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:17.447103 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:17.450707 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.451458 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.451469 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.451474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.451660 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.452048 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.452055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.452057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.452600 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:17.452901 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.453207 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.453463 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:58:17.454721 10831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:17.455073 11145 raft_consensus.cc:2258] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:17.457628 11145 ts_tablet_manager.cc:1905] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:17.464972 11145 ts_tablet_manager.cc:1918] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:17.465149 11145 log.cc:1196] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:17.465744 11145 ts_tablet_manager.cc:1939] T 17cae65bb34b4dfbaa9ac5288f91be33 P fd5178e9c2ca49db9638ba441d8d94cc: Deleting consensus metadata
I20260808 15:58:17.467819 10634 catalog_manager.cc:4841] TS fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): tablet 17cae65bb34b4dfbaa9ac5288f91be33 (table test-table [id=0c0f6f0beb8741ee957f9edfbba3dd66]) successfully deleted
I20260808 15:58:17.544891 11144 tablet_replica.cc:330] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: stopping tablet replica
I20260808 15:58:17.545447 11144 raft_consensus.cc:2229] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:17.545588 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10687
I20260808 15:58:17.563201 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10840
W20260808 15:58:17.563378 10636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:37009 (error 108)
I20260808 15:58:17.565806 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.565240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40686)
0808 15:58:17.565487 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.565607 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 15:58:17.566074 10636 catalog_manager.cc:4584] TS 19ffc267cc4e488ea64b1a1a2c86e1fd (127.27.254.1:37009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17cae65bb34b4dfbaa9ac5288f91be33: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
I20260808 15:58:17.578918 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.564397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.27.254.2:43291)
0808 15:58:17.564579 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.564585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.564605 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:58:17.564676 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.564683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.564800 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.564920 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:17.566796 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.567616 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.568938 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.569399 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.569581 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.572545 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:17.572556 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:17.572569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:17.572608 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:17.575731 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.576443 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.576449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.576452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.576590 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.576984 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.576991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.576994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.577335 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:17.578006 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.578297 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.578520 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:58:17.579684 10966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:17.619676 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.619221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40700)
0808 15:58:17.619421 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.619538 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:58:17.620654 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.620321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40716)
0808 15:58:17.620461 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.620547 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 15:58:17.681041 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.680533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40718)
0808 15:58:17.680753 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.680871 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 15:58:17.682140 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.681740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40732)
0808 15:58:17.681896 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.681974 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 15:58:17.692469 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10840
I20260808 15:58:17.710713 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10974
I20260808 15:58:17.724449 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.711514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.27.254.3:44089)
0808 15:58:17.711721 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:17.711728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:17.711748 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:58:17.711938 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:17.711950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:17.712014 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:17.712119 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:17.713452 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.714176 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.715544 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:17.715935 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:17.716064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:17.719137 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:17.719151 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:17.719167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:17.719204 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:17.721746 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.722419 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.722426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.722428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.722570 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:17.722952 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:17.722957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:17.722961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:17.723298 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:17.723599 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:17.723906 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 15:58:17.724117 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:58:17.725176 11098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:17.763437 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.762859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40736)
0808 15:58:17.763106 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.763218 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 15:58:17.764568 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.764127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40748)
0808 15:58:17.764356 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.764440 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 15:58:17.814898 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10974
I20260808 15:58:17.833124 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 10616
I20260808 15:58:17.834367 10680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:17.953869 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:17.951141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37009 (local address 127.0.0.1:40760)
0808 15:58:17.953461 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:17.953604 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:37009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2136}
I20260808 15:58:17.954762 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 10616
I20260808 15:58:17.976291 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36521
--webserver_interface=127.27.254.62
--webserver_port=36819
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:18.295451 11151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:18.295710 11151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:18.295837 11151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:18.302718 11151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:18.302824 11151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:18.302939 11151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:18.303038 11151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:18.303102 11151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:18.312155 11151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=36819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:18.313304 11151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:18.314560 11151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:18.325737 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:18.325831 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:18.325748 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:18.326673 11151 server_base.cc:1034] running on GCE node
I20260808 15:58:18.327430 11151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:18.328715 11151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:18.329978 11151 hybrid_clock.cc:648] HybridClock initialized: now 1786204698329926 us; error 34 us; skew 500 ppm
I20260808 15:58:18.330354 11151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:18 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:18.333455 11151 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:18.335165 11151 webserver.cc:460] Webserver started at http://127.27.254.62:36819/ using document root <none> and password file <none>
I20260808 15:58:18.335783 11151 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:18.335884 11151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:18.340924 11151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260808 15:58:18.343670 11167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:18.344957 11151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:58:18.345088 11151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "9c2f3b6d642549a1b7e84c8d1abf89da"
format_stamp: "Formatted at 2026-08-08 15:58:12 on dist-test-slave-srgv"
server_key: "54225eba4446fca2d18e594ea2e07034"
server_key_iv: "824ae82b08600bc55ab6c1783f972e76"
server_key_version: "encryptionkey@0"
I20260808 15:58:18.346067 11151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:18.361760 11151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:18.365432 11151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:18.365636 11151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:18.383999 11151 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:36521
I20260808 15:58:18.384068 11218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:36521 every 8 connection(s)
I20260808 15:58:18.385494 11151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:18.388912 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11151
I20260808 15:58:18.389984 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:37009
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:36521
--webserver_port=36201
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:58:18.393690 11219 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:58:18.398828 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap starting.
I20260808 15:58:18.418627 11219 log.cc:824] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:18.435273 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:58:18.435840 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da: Bootstrap complete.
I20260808 15:58:18.444659 11219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:18.445570 11219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2f3b6d642549a1b7e84c8d1abf89da, State: Initialized, Role: FOLLOWER
I20260808 15:58:18.446228 11219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:18.446463 11219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:58:18.446576 11219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:58:18.446758 11219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 2 FOLLOWER]: Advancing to term 3
I20260808 15:58:18.450860 11219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:18.451345 11219 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2f3b6d642549a1b7e84c8d1abf89da; no voters:
I20260808 15:58:18.452071 11219 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 15:58:18.452207 11223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 3 FOLLOWER]: Leader election won for term 3
I20260808 15:58:18.453223 11223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [term 3 LEADER]: Becoming Leader. State: Replica: 9c2f3b6d642549a1b7e84c8d1abf89da, State: Running, Role: LEADER
I20260808 15:58:18.453879 11223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } }
I20260808 15:58:18.454229 11219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:18.456462 11225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2f3b6d642549a1b7e84c8d1abf89da. Latest consensus state: current_term: 3 leader_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } } }
I20260808 15:58:18.456795 11225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:18.457322 11224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2f3b6d642549a1b7e84c8d1abf89da" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36521 } } }
I20260808 15:58:18.457624 11224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2f3b6d642549a1b7e84c8d1abf89da [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:18.466082 11228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:18.476449 11228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:18.478446 11228 catalog_manager.cc:1259] Loaded cluster ID: cd43036ac2a44d01994f8453636005d9
I20260808 15:58:18.478631 11228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 15:58:18.481287 11228 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f985a6f49ad google::LogMessage::Fail() at ??:0
@ 0x7f985a6f788a google::LogMessage::SendToLog() at ??:0
@ 0x7f985a6f44de google::LogMessage::Flush() at ??:0
@ 0x7f985a6f5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9870127512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9870124976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f98701ac610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f98701ac439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f985d340345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f985d34391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f985d343799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f985d31eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f98605f86db start_thread at ??:0
@ 0x7f985958e61f clone at ??:0
W20260808 15:58:18.784713 11221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:18.785069 11221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:18.785228 11221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:18.792629 11221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:18.792738 11221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:18.792948 11221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:58:18.802881 11221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:37009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=36201
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:18.804127 11221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:18.805555 11221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:18.818282 11260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:18.818246 11261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:18.818698 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:18.819132 11221 server_base.cc:1034] running on GCE node
I20260808 15:58:18.819772 11221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:18.820655 11221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:18.821877 11221 hybrid_clock.cc:648] HybridClock initialized: now 1786204698821824 us; error 45 us; skew 500 ppm
I20260808 15:58:18.822167 11221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:18 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:18.825340 11221 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:58:18.826875 11221 webserver.cc:460] Webserver started at http://127.27.254.1:36201/ using document root <none> and password file <none>
I20260808 15:58:18.827479 11221 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:18.827575 11221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:18.832602 11221 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:58:18.835392 11270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:18.836664 11221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:58:18.836830 11221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "19db8403f67d4adf35834ba08472f404"
server_key_iv: "48c5cad7cc593a091b365b570f27ff56"
server_key_version: "encryptionkey@0"
I20260808 15:58:18.837814 11221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:18.854187 11221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:18.858428 11221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:18.858686 11221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:18.859602 11221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:18.861271 11277 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 15:58:18.866674 11221 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 15:58:18.866770 11221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260808 15:58:18.866937 11221 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 15:58:18.869903 11221 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 15:58:18.869987 11221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:18.870317 11277 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Bootstrap starting.
I20260808 15:58:18.896873 11277 log.cc:824] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:18.912660 11277 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:58:18.913558 11277 tablet_bootstrap.cc:492] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Bootstrap complete.
I20260808 15:58:18.915109 11277 ts_tablet_manager.cc:1397] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.017s
I20260808 15:58:18.916162 11221 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:37009
I20260808 15:58:18.916206 11385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:37009 every 8 connection(s)
I20260808 15:58:18.917553 11221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:58:18.922564 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:18.920394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:48997)
0808 15:58:18.921322 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:18.921680 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":429,"threads_started":1}
W20260808 15:58:18.923125 11386 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:36521 (0 consecutive failures): Network error: Failed to ping master at 127.27.254.62:36521: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
I20260808 15:58:18.923988 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:18.923675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:58677)
0808 15:58:18.923803 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:18.923879 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:58:18.923378 11277 raft_consensus.cc:348] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:18.924621 11277 raft_consensus.cc:729] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ffc267cc4e488ea64b1a1a2c86e1fd, State: Initialized, Role: FOLLOWER
I20260808 15:58:18.925026 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:18.924577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:56579)
0808 15:58:18.924781 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:18.924871 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260808 15:58:18.925307 11386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:58:18.925282 11277 consensus_queue.cc:260] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:18.926714 11277 ts_tablet_manager.cc:1428] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260808 15:58:18.926769 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:18.926398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:48931)
0808 15:58:18.926552 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:18.926634 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:58:18.927512 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11221
I20260808 15:58:18.927606 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:18.927299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:51791)
0808 15:58:18.927441 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:18.927506 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:58:18.928207 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:43291
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:36521
--webserver_port=33415
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:19.250267 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:19.250522 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:19.250612 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:19.257551 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:19.257650 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:19.257756 11390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:58:19.267066 11390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:43291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=33415
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:19.268263 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:19.269912 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:19.282564 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:19.282611 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:19.282531 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:19.283291 11390 server_base.cc:1034] running on GCE node
I20260808 15:58:19.283871 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:19.284745 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:19.285975 11390 hybrid_clock.cc:648] HybridClock initialized: now 1786204699285932 us; error 54 us; skew 500 ppm
I20260808 15:58:19.286290 11390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:19 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204699, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:19.292663 11390 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:58:19.294075 11390 webserver.cc:460] Webserver started at http://127.27.254.2:33415/ using document root <none> and password file <none>
I20260808 15:58:19.294749 11390 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:19.294909 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:19.299958 11390 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 15:58:19.302678 11405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:19.303949 11390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:19.304121 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "25ed9653d16a4dcba9808d07e4ac8c99"
format_stamp: "Formatted at 2026-08-08 15:58:13 on dist-test-slave-srgv"
server_key: "d7908185c1925a527407bfa72a6b7c50"
server_key_iv: "d41bb1ff0f91e7f89e98622feff67ca7"
server_key_version: "encryptionkey@0"
I20260808 15:58:19.305070 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:19.330128 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:19.333913 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:19.334177 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:19.335264 11390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:19.336520 11390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:19.336643 11390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:19.336768 11390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:19.336862 11390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:19.372866 11390 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:43291
I20260808 15:58:19.372952 11518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:43291 every 8 connection(s)
I20260808 15:58:19.374473 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:58:19.379110 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.377356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:59525)
0808 15:58:19.378080 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.378340 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":268,"threads_started":1}
W20260808 15:58:19.379690 11519 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:36521 (0 consecutive failures): Network error: Failed to ping master at 127.27.254.62:36521: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
I20260808 15:58:19.380389 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.380112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:56841)
0808 15:58:19.380231 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.380292 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 15:58:19.381224 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.380828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:51285)
0808 15:58:19.380987 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.381075 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:58:19.381340 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11390
W20260808 15:58:19.381477 11519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:58:19.382005 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:44089
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:36521
--webserver_port=42243
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:19.705191 11522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:19.705533 11522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:19.705705 11522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:19.713069 11522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:19.713162 11522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:19.713349 11522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:58:19.722669 11522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:44089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=42243
--tserver_master_addrs=127.27.254.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:19.723893 11522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:19.725201 11522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:19.737718 11527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:19.737778 11528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:19.737757 11530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:19.738596 11522 server_base.cc:1034] running on GCE node
I20260808 15:58:19.739180 11522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:19.740149 11522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:19.741371 11522 hybrid_clock.cc:648] HybridClock initialized: now 1786204699741330 us; error 55 us; skew 500 ppm
I20260808 15:58:19.741719 11522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:19 dist-test-slave-srgv krb5kdc[10059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204699, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:19.744700 11522 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:58:19.746085 11522 webserver.cc:460] Webserver started at http://127.27.254.3:42243/ using document root <none> and password file <none>
I20260808 15:58:19.746737 11522 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:19.746881 11522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:19.751832 11522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:58:19.754372 11537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:19.755679 11522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:58:19.755858 11522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "fd5178e9c2ca49db9638ba441d8d94cc"
format_stamp: "Formatted at 2026-08-08 15:58:14 on dist-test-slave-srgv"
server_key: "2916454aebfc7838cfc3c24e83823136"
server_key_iv: "17169c7d8d0c348df59565b5eef65398"
server_key_version: "encryptionkey@0"
I20260808 15:58:19.756803 11522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:19.770903 11522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:19.774761 11522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:19.775041 11522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:19.775995 11522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:19.777275 11522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:19.777405 11522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:19.777570 11522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:19.777661 11522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:19.810356 11522 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:44089
I20260808 15:58:19.810442 11651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:44089 every 8 connection(s)
I20260808 15:58:19.811786 11522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:58:19.813727 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11522
I20260808 15:58:19.816964 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.815055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:51947)
0808 15:58:19.815914 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.816130 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":402,"threads_started":1}
W20260808 15:58:19.817704 11652 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:36521 (0 consecutive failures): Network error: Failed to ping master at 127.27.254.62:36521: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
I20260808 15:58:19.818413 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.818132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:57499)
0808 15:58:19.818254 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.818316 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 15:58:19.819133 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.818851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:38511)
0808 15:58:19.818966 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.819033 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 15:58:19.819350 11652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:58:19.929067 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:19.928353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:58979)
0808 15:58:19.928780 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:19.928854 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260808 15:58:20.382709 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.381991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:53801)
0808 15:58:20.382405 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:20.382499 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260808 15:58:20.453923 11658 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:20.454258 11658 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:20.455721 11658 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
Aug 08 15:58:20 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
Aug 08 15:58:20 dist-test-slave-srgv krb5kdc[10059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204698, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:58:20.477089 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.457148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:38955 (local address 127.27.254.3:44089)
0808 15:58:20.458601 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:20.458611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:20.458640 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:58:20.460179 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:20.460188 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:20.460524 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:20.460857 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:20.462880 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.463983 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.465701 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.466282 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.466463 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:20.470125 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:20.470133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:20.470145 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:20.470177 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:20.473542 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:20.474329 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:20.474339 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:20.474349 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:20.474504 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:20.474950 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:20.474955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:20.474960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:20.475411 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:20.475611 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:20.476371 (+ 760us) server_negotiation.cc:299] Negotiation successful
0808 15:58:20.476672 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":1208,"threads_started":1}
I20260808 15:58:20.477089 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.456174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:54007 (local address 127.27.254.2:43291)
0808 15:58:20.458347 (+ 2173us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:20.458359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:20.458392 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:58:20.460017 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:20.460026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:20.460397 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:20.460731 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:20.462877 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.464007 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.465691 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.466273 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.466473 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:20.470377 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:20.470384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:20.470395 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:20.470432 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:20.473597 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:20.474348 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:20.474355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:20.474360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:20.474495 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:20.474964 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:20.474968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:20.474970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:20.475411 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:20.475612 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:20.476371 (+ 759us) server_negotiation.cc:299] Negotiation successful
0808 15:58:20.476701 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":1843,"threads_started":1}
I20260808 15:58:20.477113 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.456039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:43291 (local address 127.27.254.1:54007)
0808 15:58:20.456567 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:20.456601 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:20.457211 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:20.460865 (+ 3654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:20.460875 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:20.461863 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:20.462668 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:20.462687 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.464175 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.464179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:20.465561 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:20.465573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.466476 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.466482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:20.466765 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:20.467580 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:20.467596 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:20.470114 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:20.473724 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:20.473729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:20.473734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:20.474204 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:20.474611 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:20.474615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:20.474619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:20.474832 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:20.475564 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:20.475571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:20.475882 (+ 311us) client_negotiation.cc:769] Sending connection context
0808 15:58:20.476128 (+ 246us) client_negotiation.cc:240] Negotiation successful
0808 15:58:20.476617 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":115,"spinlock_wait_cycles":19200,"thread_start_us":283,"threads_started":1}
I20260808 15:58:20.477219 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.456770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:44089 (local address 127.27.254.1:38955)
0808 15:58:20.458069 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:20.458086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:20.458188 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:20.460883 (+ 2695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:20.460888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:20.461874 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:20.462668 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:20.462688 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.464185 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.464189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:20.465563 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:20.465575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:20.466536 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:20.466543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:20.466766 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:20.467516 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:20.467541 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:20.469840 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:20.473676 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:20.473685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:20.473698 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:20.474188 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:20.474620 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:20.474625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:20.474628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:20.474831 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:20.475565 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:20.475572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:20.475882 (+ 310us) client_negotiation.cc:769] Sending connection context
0808 15:58:20.476128 (+ 246us) client_negotiation.cc:240] Negotiation successful
0808 15:58:20.476780 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":422,"spinlock_wait_cycles":27264,"thread_start_us":978,"threads_started":1}
W20260808 15:58:20.480278 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
W20260808 15:58:20.480593 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:20.480720 11272 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd; no voters: 25ed9653d16a4dcba9808d07e4ac8c99, fd5178e9c2ca49db9638ba441d8d94cc
I20260808 15:58:20.481118 11658 raft_consensus.cc:2726] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:58:20.820717 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.819828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:41665)
0808 15:58:20.820366 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:20.820503 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":297,"threads_started":1}
I20260808 15:58:20.930469 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:20.930071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:58441)
0808 15:58:20.930229 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:20.930327 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:58:21.384229 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:21.383518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:49251)
0808 15:58:21.383997 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:21.384085 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
I20260808 15:58:21.822295 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:21.821652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:33847)
0808 15:58:21.822072 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:21.822150 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260808 15:58:21.932137 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:21.931389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:58245)
0808 15:58:21.931880 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:21.931966 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":213,"threads_started":1}
I20260808 15:58:22.245621 11667 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:22.245808 11667 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:22.246353 11667 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
W20260808 15:58:22.247364 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
W20260808 15:58:22.247613 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:22.247781 11272 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd; no voters: 25ed9653d16a4dcba9808d07e4ac8c99, fd5178e9c2ca49db9638ba441d8d94cc
I20260808 15:58:22.248027 11667 raft_consensus.cc:2726] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:58:22.385778 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:22.385063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:50847)
0808 15:58:22.385530 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:22.385609 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260808 15:58:22.823845 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:22.823052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:43431)
0808 15:58:22.823598 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:22.823680 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":334,"threads_started":1}
I20260808 15:58:22.933972 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:22.933130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:54359)
0808 15:58:22.933624 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:22.933706 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":295,"threads_started":1}
I20260808 15:58:23.387324 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:23.386663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:52911)
0808 15:58:23.387088 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:23.387162 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":228,"threads_started":1}
I20260808 15:58:23.825568 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:23.824735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:51051)
0808 15:58:23.825204 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:23.825300 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260808 15:58:23.935765 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:23.935087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:55803)
0808 15:58:23.935505 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:23.935584 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260808 15:58:24.389003 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:24.388339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:56997)
0808 15:58:24.388766 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:24.388845 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260808 15:58:24.409653 11675 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:24.409914 11675 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:24.410542 11675 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
W20260808 15:58:24.411543 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
W20260808 15:58:24.411722 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:24.411806 11272 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd; no voters: 25ed9653d16a4dcba9808d07e4ac8c99, fd5178e9c2ca49db9638ba441d8d94cc
I20260808 15:58:24.412057 11675 raft_consensus.cc:2726] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:58:24.827119 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:24.826478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:49527)
0808 15:58:24.826883 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:24.826965 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
I20260808 15:58:24.937284 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:24.936600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:60287)
0808 15:58:24.937042 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:24.937128 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260808 15:58:25.390568 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:25.389881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:44061)
0808 15:58:25.390308 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:25.390391 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
I20260808 15:58:25.828711 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:25.828012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:50335)
0808 15:58:25.828464 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:25.828544 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260808 15:58:25.938820 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:25.938133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:46657)
0808 15:58:25.938555 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:25.938631 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260808 15:58:26.134058 11681 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:26.134274 11681 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:26.134996 11681 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
W20260808 15:58:26.135897 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
W20260808 15:58:26.136198 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:26.136307 11272 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd; no voters: 25ed9653d16a4dcba9808d07e4ac8c99, fd5178e9c2ca49db9638ba441d8d94cc
I20260808 15:58:26.136550 11681 raft_consensus.cc:2726] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:58:26.392120 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:26.391358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:52551)
0808 15:58:26.391869 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:26.391953 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":245,"threads_started":1}
I20260808 15:58:26.830277 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:26.829545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:57279)
0808 15:58:26.830024 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:26.830099 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":238,"threads_started":1}
I20260808 15:58:26.940282 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:26.939631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:53989)
0808 15:58:26.940043 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:26.940117 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260808 15:58:27.393795 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:27.393006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:48653)
0808 15:58:27.393522 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:27.393622 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":274,"threads_started":1}
I20260808 15:58:27.831864 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:27.831194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:36525)
0808 15:58:27.831600 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:27.831675 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":220,"threads_started":1}
I20260808 15:58:27.941890 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:27.941125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:42691)
0808 15:58:27.941610 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:27.941691 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":259,"threads_started":1}
I20260808 15:58:28.395378 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:28.394659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:45603)
0808 15:58:28.395147 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:28.395229 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
I20260808 15:58:28.833595 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:28.832827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:36233)
0808 15:58:28.833303 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:28.833378 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260808 15:58:28.943617 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:28.942843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:57087)
0808 15:58:28.943305 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:28.943391 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260808 15:58:29.397050 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:29.396343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:39357)
0808 15:58:29.396797 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:29.396881 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260808 15:58:29.835215 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:29.834487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:54099)
0808 15:58:29.834962 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:29.835041 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260808 15:58:29.945701 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:29.944675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:58381)
0808 15:58:29.945281 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:29.945398 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":312,"threads_started":1}
I20260808 15:58:30.215713 11694 raft_consensus.cc:482] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:30.215931 11694 raft_consensus.cc:504] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ffc267cc4e488ea64b1a1a2c86e1fd" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37009 } } peers { permanent_uuid: "25ed9653d16a4dcba9808d07e4ac8c99" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 43291 } } peers { permanent_uuid: "fd5178e9c2ca49db9638ba441d8d94cc" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 44089 } }
I20260808 15:58:30.216499 11694 leader_election.cc:288] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291), fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089)
W20260808 15:58:30.217571 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 25ed9653d16a4dcba9808d07e4ac8c99 (127.27.254.2:43291): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
W20260808 15:58:30.217798 11272 leader_election.cc:341] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd5178e9c2ca49db9638ba441d8d94cc (127.27.254.3:44089): Not found: Tablet not found: 17cae65bb34b4dfbaa9ac5288f91be33
I20260808 15:58:30.217940 11272 leader_election.cc:302] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ffc267cc4e488ea64b1a1a2c86e1fd; no voters: 25ed9653d16a4dcba9808d07e4ac8c99, fd5178e9c2ca49db9638ba441d8d94cc
I20260808 15:58:30.218186 11694 raft_consensus.cc:2726] T 17cae65bb34b4dfbaa9ac5288f91be33 P 19ffc267cc4e488ea64b1a1a2c86e1fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:58:30.398744 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:30.397998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:51853)
0808 15:58:30.398495 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:30.398582 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
I20260808 15:58:30.836995 11696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:30.836262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:56715)
0808 15:58:30.836737 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:30.836834 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":217,"threads_started":1}
I20260808 15:58:30.947302 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:30.946564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:46837)
0808 15:58:30.947041 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:30.947129 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
I20260808 15:58:31.400346 11698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:31.399609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:47729)
0808 15:58:31.400073 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:31.400154 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":268,"threads_started":1}
I20260808 15:58:31.838933 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:31.838091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:60903)
0808 15:58:31.838611 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:31.838717 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260808 15:58:31.949213 11700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:31.948279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:36519)
0808 15:58:31.948863 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:31.948980 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":333,"threads_started":1}
I20260808 15:58:32.402153 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:32.401485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:35053)
0808 15:58:32.401903 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:32.401995 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":213,"threads_started":1}
I20260808 15:58:32.840654 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:32.839896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:52099)
0808 15:58:32.840344 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:32.840503 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260808 15:58:32.950870 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:32.950148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:42101)
0808 15:58:32.950601 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:32.950681 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260808 15:58:33.403803 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:33.403072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:59603)
0808 15:58:33.403519 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:33.403600 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260808 15:58:33.842512 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:33.841720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:51531)
0808 15:58:33.842164 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:33.842244 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260808 15:58:33.953147 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:33.952188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.1:52773)
0808 15:58:33.952861 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:33.952965 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":333,"threads_started":1}
I20260808 15:58:34.405827 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:34.404868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.2:38987)
0808 15:58:34.405478 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:34.405593 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":305,"threads_started":1}
I20260808 15:58:34.817951 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 11221
I20260808 15:58:34.833545 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:34.819077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.27.254.1:37009)
0808 15:58:34.819635 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:34.819645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:34.819689 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:58:34.819788 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:34.819799 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:34.820075 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:34.820278 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:34.822454 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:34.823412 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:34.824629 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:34.825241 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:34.825363 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:34.828183 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:34.828193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:34.828206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:34.828233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:34.830945 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:34.831592 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:34.831599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:34.831601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:34.831718 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:34.832023 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:34.832026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:34.832028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:34.832356 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:34.832597 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:34.832904 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 15:58:34.833108 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":307,"threads_started":1}
I20260808 15:58:34.834829 11381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:34.844259 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:34.843496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:36521 (local address 127.27.254.3:49533)
0808 15:58:34.843997 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:34.844118 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":285,"threads_started":1}
I20260808 15:58:34.926620 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 11221
I20260808 15:58:34.946327 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 11390
I20260808 15:58:34.959466 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:34.947091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.27.254.2:43291)
0808 15:58:34.947653 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:34.947658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:34.947674 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:58:34.947730 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:34.947733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:34.947792 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:34.947887 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:34.949211 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:34.950081 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:34.951324 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:34.951740 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:34.951829 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:34.954549 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:34.954554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:34.954557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:34.954605 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:34.957021 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:34.957758 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:34.957762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:34.957764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:34.957901 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:34.958213 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:34.958216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:34.958218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:34.958457 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:34.958642 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:34.958900 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 15:58:34.959059 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":337,"threads_started":1}
I20260808 15:58:34.960294 11514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:35.043905 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 11390
I20260808 15:58:35.061192 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 11522
I20260808 15:58:35.074267 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:35.061996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.27.254.3:44089)
0808 15:58:35.062552 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:35.062557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:35.062575 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:58:35.062630 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:35.062634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:35.062701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:35.062798 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:35.064135 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:35.064893 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:35.066141 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:35.066563 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:35.066656 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:35.069130 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:35.069134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:35.069138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:35.069166 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:35.071821 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:35.072570 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:35.072574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:35.072576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:35.072721 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:35.073059 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:35.073062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:35.073063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:35.073322 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:35.073609 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:35.073825 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 15:58:35.073981 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":308,"threads_started":1}
I20260808 15:58:35.074928 11646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:35.159469 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 11522
2026-08-08T15:58:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24339 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:58:35 dist-test-slave-srgv krb5kdc[11717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:58:35 dist-test-slave-srgv krb5kdc[11717](info): set up 2 sockets
Aug 08 15:58:35 dist-test-slave-srgv krb5kdc[11717](info): commencing operation
krb5kdc: starting...
W20260808 15:58:36.264005 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:58:36 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:58:36.509915 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33817
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:36.836541 11731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:36.836805 11731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:36.836990 11731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:36.844115 11731 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:58:36.844209 11731 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:58:36.844282 11731 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:58:36.844362 11731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:36.844450 11731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:36.844574 11731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:36.844636 11731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:36.844743 11731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:36.854542 11731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:36.855983 11731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:36.857370 11731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:36.869769 11736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:36.869856 11737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:36.869812 11739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:36.870569 11731 server_base.cc:1034] running on GCE node
I20260808 15:58:36.871308 11731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:36.872632 11731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:36.873862 11731 hybrid_clock.cc:648] HybridClock initialized: now 1786204716873792 us; error 30 us; skew 500 ppm
I20260808 15:58:36.874225 11731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:36 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204716, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:36.877562 11731 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:36.879037 11731 webserver.cc:460] Webserver started at http://127.27.254.62:39447/ using document root <none> and password file <none>
I20260808 15:58:36.879724 11731 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:36.879873 11731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:36.880249 11731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:36.883284 11731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "bc7a9df785204bd4b5be5dac922ddd09"
format_stamp: "Formatted at 2026-08-08 15:58:36 on dist-test-slave-srgv"
server_key: "4be02cc4c9e62e562cca4dc9d4242db5"
server_key_iv: "92abde8d060e32bc0f94cf50822b435e"
server_key_version: "encryptionkey@0"
I20260808 15:58:36.884069 11731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "bc7a9df785204bd4b5be5dac922ddd09"
format_stamp: "Formatted at 2026-08-08 15:58:36 on dist-test-slave-srgv"
server_key: "4be02cc4c9e62e562cca4dc9d4242db5"
server_key_iv: "92abde8d060e32bc0f94cf50822b435e"
server_key_version: "encryptionkey@0"
I20260808 15:58:36.889159 11731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:58:36.892629 11746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:36.894042 11731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:58:36.894224 11731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "bc7a9df785204bd4b5be5dac922ddd09"
format_stamp: "Formatted at 2026-08-08 15:58:36 on dist-test-slave-srgv"
server_key: "4be02cc4c9e62e562cca4dc9d4242db5"
server_key_iv: "92abde8d060e32bc0f94cf50822b435e"
server_key_version: "encryptionkey@0"
I20260808 15:58:36.894440 11731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:36.910498 11731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:36.914582 11731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:36.914870 11731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:36.933202 11731 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33817
I20260808 15:58:36.933180 11798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33817 every 8 connection(s)
I20260808 15:58:36.934525 11731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:36.938064 11799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:36.943624 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11731
I20260808 15:58:36.943859 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:58:36.944512 28664 external_mini_cluster.cc:1428] Setting key 61ca06eee3cc047c06e067e3fe0e079f
I20260808 15:58:36.949678 11799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap starting.
I20260808 15:58:36.953434 11799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:36.954720 11799 log.cc:824] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:36.958050 11799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: No bootstrap required, opened a new log
Aug 08 15:58:36 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:36.966285 11799 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:36.966684 11799 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:36.966845 11799 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7a9df785204bd4b5be5dac922ddd09, State: Initialized, Role: FOLLOWER
I20260808 15:58:36.967423 11799 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:36.967689 11799 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:58:36.967844 11799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:58:36.968042 11799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:36.969730 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:36.946608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.27.254.62:33817)
0808 15:58:36.947477 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:36.947494 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:36.947542 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:58:36.949948 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:36.949979 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:36.950484 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:36.950923 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:36.952371 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:36.954443 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:36.955846 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:36.956553 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:36.956697 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:36.961738 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:36.961747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:36.961761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:36.961805 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:36.964964 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:36.965940 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:36.965953 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:36.965959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:36.966140 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:36.966645 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:36.966650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:36.966653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:36.967418 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:36.967732 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:36.968149 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 15:58:36.968579 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":393,"threads_started":1}
I20260808 15:58:36.970662 11799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:36.971252 11799 leader_election.cc:302] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7a9df785204bd4b5be5dac922ddd09; no voters:
I20260808 15:58:36.971910 11799 leader_election.cc:288] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:58:36.972021 11804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:36.973680 11804 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 LEADER]: Becoming Leader. State: Replica: bc7a9df785204bd4b5be5dac922ddd09, State: Running, Role: LEADER
I20260808 15:58:36.974081 11799 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:36.974298 11804 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:36.978430 11805 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7a9df785204bd4b5be5dac922ddd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } } }
I20260808 15:58:36.978434 11806 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7a9df785204bd4b5be5dac922ddd09. Latest consensus state: current_term: 1 leader_uuid: "bc7a9df785204bd4b5be5dac922ddd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } } }
I20260808 15:58:36.978856 11805 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:36.978857 11806 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:36.980625 11815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:36.985486 11815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:36.994262 11815 catalog_manager.cc:1347] Generated new cluster ID: 1b05373bf6c048ed95da5efc2b794c78
I20260808 15:58:36.994374 11815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:58:37.009228 11815 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:58:37.010600 11815 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:58:37.018155 11815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Generated new TSK 0
I20260808 15:58:37.018981 11815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:58:37.085495 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33817
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:37.410941 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:37.411273 11825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:37.411443 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:37.418745 11825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:37.418913 11825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:37.419099 11825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:58:37.428735 11825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:37.430186 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:37.431593 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:37.444674 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:37.444694 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:37.444860 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:37.445919 11825 server_base.cc:1034] running on GCE node
I20260808 15:58:37.446599 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:37.447515 11825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:37.448716 11825 hybrid_clock.cc:648] HybridClock initialized: now 1786204717448665 us; error 38 us; skew 500 ppm
I20260808 15:58:37.449064 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:37 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:37.452212 11825 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:58:37.453732 11825 webserver.cc:460] Webserver started at http://127.27.254.1:41403/ using document root <none> and password file <none>
I20260808 15:58:37.454416 11825 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:37.454553 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:37.454877 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:37.457842 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "359ec443e3ce9f8efc299d32fc1cf884"
server_key_iv: "0221a19532f889d2388e07ce8f488f3b"
server_key_version: "encryptionkey@0"
I20260808 15:58:37.458714 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "359ec443e3ce9f8efc299d32fc1cf884"
server_key_iv: "0221a19532f889d2388e07ce8f488f3b"
server_key_version: "encryptionkey@0"
I20260808 15:58:37.463733 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260808 15:58:37.467069 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:37.468443 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:58:37.468619 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "359ec443e3ce9f8efc299d32fc1cf884"
server_key_iv: "0221a19532f889d2388e07ce8f488f3b"
server_key_version: "encryptionkey@0"
I20260808 15:58:37.468825 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:37.493741 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:37.497799 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:37.498129 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:37.499271 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:37.500581 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:37.500686 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:37.500844 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:37.500921 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:37.534799 11825 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:44397
I20260808 15:58:37.534896 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:44397 every 8 connection(s)
I20260808 15:58:37.536111 11825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:58:37.539688 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11825
I20260808 15:58:37.539922 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:58:37.540642 28664 external_mini_cluster.cc:1428] Setting key 1fb4ee69c9e4b5a4d603b718d636d2ae
Aug 08 15:58:37 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:37.557075 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:37.538810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:60035 (local address 127.27.254.62:33817)
0808 15:58:37.539296 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:37.539302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:37.540427 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 15:58:37.542525 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:37.542528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:37.542609 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:37.542724 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:37.544247 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:37.545436 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:37.546972 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:37.547688 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:37.547850 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:37.551455 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:37.551460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:37.551463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:37.551495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:37.554069 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:37.554800 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:37.554806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:37.554810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:37.554936 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:37.555370 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:37.555375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:37.555378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:37.555632 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:37.555788 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:37.556476 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 15:58:37.556713 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260808 15:58:37.558148 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:37.539395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:60035)
0808 15:58:37.540237 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:37.540283 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:37.542234 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:37.542981 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:37.542990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:37.543491 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:37.544062 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:37.544080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:37.545604 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:37.545608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:37.546801 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:37.546813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:37.547903 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:37.547908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:37.548093 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:37.548784 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:37.548806 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:37.551228 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:37.554209 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:37.554217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:37.554233 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:37.554679 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:37.555048 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:37.555053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:37.555056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:37.555248 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:37.555767 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:37.555774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:37.556100 (+ 326us) client_negotiation.cc:769] Sending connection context
0808 15:58:37.556366 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:58:37.556894 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260808 15:58:37.559811 11954 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33817
I20260808 15:58:37.560125 11954 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:37.560818 11954 heartbeater.cc:511] Master 127.27.254.62:33817 requested a full tablet report, sending...
I20260808 15:58:37.562916 11763 ts_manager.cc:194] Registered new tserver with Master: 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397)
I20260808 15:58:37.564358 11763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:60035
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:58:37.594928 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33817
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:37.919293 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:37.919584 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:37.919792 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:37.927107 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:37.927208 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:37.927317 11960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:58:37.939850 11960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:37.941268 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:37.942795 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:37.957994 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:37.957967 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:37.957947 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:37.958841 11960 server_base.cc:1034] running on GCE node
I20260808 15:58:37.959460 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:37.960382 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:37.961624 11960 hybrid_clock.cc:648] HybridClock initialized: now 1786204717961555 us; error 59 us; skew 500 ppm
I20260808 15:58:37.961968 11960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:37 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:37.970054 11960 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:58:37.971446 11960 webserver.cc:460] Webserver started at http://127.27.254.2:38827/ using document root <none> and password file <none>
I20260808 15:58:37.972154 11960 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:37.972304 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:37.972676 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:37.975667 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "a4fb09f6ed564c9c82b663e3d3691bf7"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "4fdd6d3c52a5f76051a4d42b199c9df7"
server_key_iv: "999a77a6259b27a160c395b94d52eb0a"
server_key_version: "encryptionkey@0"
I20260808 15:58:37.976468 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "a4fb09f6ed564c9c82b663e3d3691bf7"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "4fdd6d3c52a5f76051a4d42b199c9df7"
server_key_iv: "999a77a6259b27a160c395b94d52eb0a"
server_key_version: "encryptionkey@0"
I20260808 15:58:37.981709 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:58:37.985035 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:37.986460 11960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:58:37.986654 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "a4fb09f6ed564c9c82b663e3d3691bf7"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "4fdd6d3c52a5f76051a4d42b199c9df7"
server_key_iv: "999a77a6259b27a160c395b94d52eb0a"
server_key_version: "encryptionkey@0"
I20260808 15:58:37.986893 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:38.006120 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:38.010053 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:38.010363 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:38.011320 11960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:38.012655 11960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:38.012766 11960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:38.012935 11960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:38.013013 11960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:38.047201 11960 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:40895
I20260808 15:58:38.047253 12088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:40895 every 8 connection(s)
I20260808 15:58:38.048593 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:58:38.058897 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 11960
I20260808 15:58:38.059109 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:58:38.059554 28664 external_mini_cluster.cc:1428] Setting key 65f74716788fdd4a7b8efe0133b6b7dd
Aug 08 15:58:38 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204717, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:38.068621 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.051276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:45959 (local address 127.27.254.62:33817)
0808 15:58:38.051456 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.051460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.052615 (+ 1155us) server_negotiation.cc:407] Connection header received
0808 15:58:38.054520 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.054525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.054616 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.054742 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:38.056205 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.056904 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.058513 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.059200 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.059347 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.062799 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:38.062805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:38.062809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:38.062849 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:38.065611 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.066361 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.066366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.066370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.066504 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.066958 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.066967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.066969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.067230 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:38.067390 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.068055 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.068277 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:58:38.069612 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.051722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:45959)
0808 15:58:38.052449 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.052486 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.054252 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.054969 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.054977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.055500 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:38.056033 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.056049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.057062 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.057073 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.058356 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.058367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.059430 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.059437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.059678 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.060384 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:38.060410 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:38.062574 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:38.065793 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:38.065802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:38.065816 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:38.066229 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:38.066622 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:38.066627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:38.066631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:38.066815 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:38.067361 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:38.067369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:38.067698 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.067964 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.068483 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":293,"threads_started":1}
I20260808 15:58:38.071175 12089 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33817
I20260808 15:58:38.071477 12089 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:38.072324 12089 heartbeater.cc:511] Master 127.27.254.62:33817 requested a full tablet report, sending...
I20260808 15:58:38.073781 11763 ts_manager.cc:194] Registered new tserver with Master: a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
I20260808 15:58:38.074630 11763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:45959
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:58:38.112596 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33817
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:38.439354 12094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:38.439615 12094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:38.439707 12094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:38.446537 12094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:38.446638 12094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:38.446758 12094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:58:38.456769 12094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:38.457978 12094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:38.459192 12094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:38.472234 12099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:38.472163 12100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:38.472651 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:38.472865 12094 server_base.cc:1034] running on GCE node
I20260808 15:58:38.473636 12094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:38.474606 12094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:38.475837 12094 hybrid_clock.cc:648] HybridClock initialized: now 1786204718475786 us; error 65 us; skew 500 ppm
I20260808 15:58:38.476151 12094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:38 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:38.479285 12094 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:58:38.480671 12094 webserver.cc:460] Webserver started at http://127.27.254.3:44773/ using document root <none> and password file <none>
I20260808 15:58:38.481371 12094 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:38.481585 12094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:38.481899 12094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:58:38.484687 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "be7792734edf4738a59df72d2bf1a77e"
format_stamp: "Formatted at 2026-08-08 15:58:38 on dist-test-slave-srgv"
server_key: "db40d32ff7e079122f6342b7a4669325"
server_key_iv: "d7a29ed399d0de356adf8e782dd183a4"
server_key_version: "encryptionkey@0"
I20260808 15:58:38.485488 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "be7792734edf4738a59df72d2bf1a77e"
format_stamp: "Formatted at 2026-08-08 15:58:38 on dist-test-slave-srgv"
server_key: "db40d32ff7e079122f6342b7a4669325"
server_key_iv: "d7a29ed399d0de356adf8e782dd183a4"
server_key_version: "encryptionkey@0"
I20260808 15:58:38.490886 12094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:58:38.494966 12109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:38.496272 12094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:58:38.496461 12094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "be7792734edf4738a59df72d2bf1a77e"
format_stamp: "Formatted at 2026-08-08 15:58:38 on dist-test-slave-srgv"
server_key: "db40d32ff7e079122f6342b7a4669325"
server_key_iv: "d7a29ed399d0de356adf8e782dd183a4"
server_key_version: "encryptionkey@0"
I20260808 15:58:38.496666 12094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:38.534339 12094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:38.539251 12094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:38.539583 12094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:38.540545 12094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:58:38.541924 12094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:58:38.542001 12094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:38.542083 12094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:58:38.542126 12094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:38.567425 11954 heartbeater.cc:503] Master 127.27.254.62:33817 was elected leader, sending a full tablet report...
I20260808 15:58:38.575752 12094 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37075
I20260808 15:58:38.575816 12221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37075 every 8 connection(s)
I20260808 15:58:38.577131 12094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:58:38.585749 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12094
I20260808 15:58:38.585970 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:58:38.586414 28664 external_mini_cluster.cc:1428] Setting key f16af905ddca53380549689d8e4cb90f
Aug 08 15:58:38 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204718, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:58:38.597144 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.579894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:41925 (local address 127.27.254.62:33817)
0808 15:58:38.580379 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.580384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.581156 (+ 772us) server_negotiation.cc:407] Connection header received
0808 15:58:38.582988 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.582993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.583083 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.583208 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:38.584681 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.585561 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.587318 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.587861 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.587961 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.591900 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:38.591904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:38.591908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:38.591937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:38.594606 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.595251 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.595255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.595257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.595372 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.595723 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.595726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.595728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.595975 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:38.596082 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.596701 (+ 619us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.596868 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":285,"threads_started":1}
I20260808 15:58:38.598148 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.580349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:41925)
0808 15:58:38.580997 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.581040 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.582720 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.583445 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.583453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.583941 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:58:38.584489 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.584506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.585719 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.585724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.587144 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.587158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.588096 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.588104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.588392 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.589193 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:58:38.589225 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:58:38.591710 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:58:38.594729 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:38.594735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:38.594747 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:38.595148 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:38.595461 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:58:38.595464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:58:38.595467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:58:38.595630 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:58:38.596083 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:58:38.596088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:58:38.596391 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.596614 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.596994 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
I20260808 15:58:38.599494 12223 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33817
I20260808 15:58:38.599862 12223 heartbeater.cc:464] Registering TS with master...
I20260808 15:58:38.600795 12223 heartbeater.cc:511] Master 127.27.254.62:33817 requested a full tablet report, sending...
I20260808 15:58:38.602295 11763 ts_manager.cc:194] Registered new tserver with Master: be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075)
I20260808 15:58:38.603101 11763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:41925
I20260808 15:58:38.614035 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:58:38.631228 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.618364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.27.254.62:33817)
0808 15:58:38.618519 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.618523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.618966 (+ 443us) server_negotiation.cc:407] Connection header received
0808 15:58:38.619163 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.619169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.619235 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.619338 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:38.620781 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.621573 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.623000 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.623400 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.623532 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.626106 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:38.626110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:38.626113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:38.626142 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:38.628555 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.629217 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.629224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.629227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.629348 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.629767 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.629774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.629777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.630091 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:38.630352 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.630682 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.630915 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 15:58:38.634946 11763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:58:38.641819 11763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:58:38.665519 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.653793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.27.254.1:44397)
0808 15:58:38.657068 (+ 3275us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.657078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.657118 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:58:38.657221 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.657229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.657525 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.658620 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:38.658848 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.660230 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.663242 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.664191 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.664354 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.664597 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.664979 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.665141 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3114,"thread_start_us":2966,"threads_started":1}
I20260808 15:58:38.665629 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.653452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42798)
0808 15:58:38.655593 (+ 2141us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.655639 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.655830 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.657768 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.657778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.657813 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:38.658320 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.658343 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.660382 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.660393 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.663051 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.663069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.664484 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.664492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.664725 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.664776 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.664925 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.665123 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1831,"mutex_wait_us":196,"thread_start_us":1609,"threads_started":1}
I20260808 15:58:38.665992 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.656805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.27.254.2:40895)
0808 15:58:38.658587 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.658594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.658617 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:58:38.658712 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.658720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.658931 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.659133 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:38.659698 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.661082 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.663297 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.664197 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.664362 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.664569 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.665313 (+ 744us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.665509 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":1506,"threads_started":1}
I20260808 15:58:38.666555 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.656364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40895 (local address 127.0.0.1:52552)
0808 15:58:38.657389 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.657423 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.657527 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.659143 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.659147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.659172 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:38.659512 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.659522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.661246 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.661264 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.663110 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.663123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.664424 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.664430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.664637 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.664778 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.664900 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.665119 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":151,"thread_start_us":801,"threads_started":1}
I20260808 15:58:38.668370 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.654189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37075 (local address 127.0.0.1:37920)
0808 15:58:38.656326 (+ 2137us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.656348 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.656493 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.657865 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.657871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.657898 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:38.658348 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.658357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.660156 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.660163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.663051 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.663069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.667531 (+ 4462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.667538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.667728 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.667761 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.667861 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.667990 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":89,"spinlock_wait_cycles":768,"thread_start_us":1779,"threads_started":1}
I20260808 15:58:38.668434 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.653859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.27.254.3:37075)
0808 15:58:38.656495 (+ 2636us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.656503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.656540 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:58:38.656649 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.656658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.657650 (+ 992us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.658448 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:38.658723 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.660027 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.664688 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.667323 (+ 2635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.667458 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.667671 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.667928 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.668089 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2445,"thread_start_us":3116,"threads_started":1}
I20260808 15:58:38.670264 12023 tablet_service.cc:1467] Processing CreateTablet for tablet a87e733912d64949b3db1912fd6f2653 (DEFAULT_TABLE table=test-table [id=a676286cefe24bfe81c19ceb31933133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:38.670346 11888 tablet_service.cc:1467] Processing CreateTablet for tablet a87e733912d64949b3db1912fd6f2653 (DEFAULT_TABLE table=test-table [id=a676286cefe24bfe81c19ceb31933133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:38.671610 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87e733912d64949b3db1912fd6f2653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:38.671674 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87e733912d64949b3db1912fd6f2653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:38.678664 12157 tablet_service.cc:1467] Processing CreateTablet for tablet a87e733912d64949b3db1912fd6f2653 (DEFAULT_TABLE table=test-table [id=a676286cefe24bfe81c19ceb31933133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:58:38.680151 12157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87e733912d64949b3db1912fd6f2653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:58:38.684242 12242 tablet_bootstrap.cc:492] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Bootstrap starting.
I20260808 15:58:38.684990 12243 tablet_bootstrap.cc:492] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Bootstrap starting.
I20260808 15:58:38.688016 12242 tablet_bootstrap.cc:654] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:38.688331 12243 tablet_bootstrap.cc:654] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:38.689352 12242 log.cc:824] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:38.689441 12243 log.cc:824] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:38.692461 12242 tablet_bootstrap.cc:492] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: No bootstrap required, opened a new log
I20260808 15:58:38.692742 12242 ts_tablet_manager.cc:1397] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 15:58:38.693054 12243 tablet_bootstrap.cc:492] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: No bootstrap required, opened a new log
I20260808 15:58:38.693372 12243 ts_tablet_manager.cc:1397] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260808 15:58:38.693547 12246 tablet_bootstrap.cc:492] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Bootstrap starting.
I20260808 15:58:38.697250 12246 tablet_bootstrap.cc:654] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Neither blocks nor log segments found. Creating new log.
I20260808 15:58:38.698616 12246 log.cc:824] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:38.701361 12243 raft_consensus.cc:348] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.701787 12243 raft_consensus.cc:374] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:38.701881 12246 tablet_bootstrap.cc:492] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: No bootstrap required, opened a new log
I20260808 15:58:38.701912 12243 raft_consensus.cc:729] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fb09f6ed564c9c82b663e3d3691bf7, State: Initialized, Role: FOLLOWER
I20260808 15:58:38.702134 12246 ts_tablet_manager.cc:1397] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 15:58:38.702548 12243 consensus_queue.cc:260] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.703701 12089 heartbeater.cc:503] Master 127.27.254.62:33817 was elected leader, sending a full tablet report...
I20260808 15:58:38.704185 12243 ts_tablet_manager.cc:1428] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260808 15:58:38.703271 12242 raft_consensus.cc:348] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.704538 12242 raft_consensus.cc:374] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:38.704663 12242 raft_consensus.cc:729] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e19dc8640354a5a86a3c8a3b7627a67, State: Initialized, Role: FOLLOWER
I20260808 15:58:38.705241 12242 consensus_queue.cc:260] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.706892 12242 ts_tablet_manager.cc:1428] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260808 15:58:38.712352 12246 raft_consensus.cc:348] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.712725 12246 raft_consensus.cc:374] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:58:38.712798 12246 raft_consensus.cc:729] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7792734edf4738a59df72d2bf1a77e, State: Initialized, Role: FOLLOWER
I20260808 15:58:38.713212 12246 consensus_queue.cc:260] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.714221 12223 heartbeater.cc:503] Master 127.27.254.62:33817 was elected leader, sending a full tablet report...
I20260808 15:58:38.714811 12246 ts_tablet_manager.cc:1428] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260808 15:58:38.760738 12248 raft_consensus.cc:482] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:58:38.761060 12248 raft_consensus.cc:504] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.762812 12248 leader_election.cc:288] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397), be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075)
I20260808 15:58:38.769835 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.763469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:55185 (local address 127.27.254.3:37075)
0808 15:58:38.763713 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.763721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.764307 (+ 586us) server_negotiation.cc:407] Connection header received
0808 15:58:38.764530 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.764534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.764594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.764704 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:38.765375 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.766143 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.768042 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.768767 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.768965 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.769040 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.769394 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.769532 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:58:38.769840 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.763257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:59893 (local address 127.27.254.1:44397)
0808 15:58:38.763503 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.763511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.764295 (+ 784us) server_negotiation.cc:407] Connection header received
0808 15:58:38.764542 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.764546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.764602 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.764702 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:58:38.765311 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.766104 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.768047 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.768796 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.768977 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.769054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.769397 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.769540 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 15:58:38.770076 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.763093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.27.254.2:59893)
0808 15:58:38.764151 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.764170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.764387 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.764716 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.764720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.764757 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:38.765148 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.765157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.766234 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.766241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.767891 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.767902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.769029 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.769034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.769234 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.769256 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.769370 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.769516 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":126,"spinlock_wait_cycles":26112,"thread_start_us":640,"threads_started":1}
I20260808 15:58:38.770378 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.763265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37075 (local address 127.27.254.2:55185)
0808 15:58:38.764190 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:38.764208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:58:38.764385 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:58:38.764715 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:58:38.764719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:58:38.764750 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:58:38.765213 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.765223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.766293 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.766303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.767889 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:58:38.767900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.768995 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.769002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:58:38.769237 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.769266 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:58:38.769367 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:58:38.769512 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":91,"spinlock_wait_cycles":8320,"thread_start_us":637,"threads_started":1}
I20260808 15:58:38.771261 11908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87e733912d64949b3db1912fd6f2653" candidate_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" is_pre_election: true
I20260808 15:58:38.771389 12177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87e733912d64949b3db1912fd6f2653" candidate_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7792734edf4738a59df72d2bf1a77e" is_pre_election: true
I20260808 15:58:38.771806 11908 raft_consensus.cc:2454] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4fb09f6ed564c9c82b663e3d3691bf7 in term 0.
I20260808 15:58:38.771981 12177 raft_consensus.cc:2454] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4fb09f6ed564c9c82b663e3d3691bf7 in term 0.
I20260808 15:58:38.772593 11976 leader_election.cc:302] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e19dc8640354a5a86a3c8a3b7627a67, a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e; no voters:
I20260808 15:58:38.773083 12248 raft_consensus.cc:2781] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:58:38.773232 12248 raft_consensus.cc:482] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:58:38.773360 12248 raft_consensus.cc:3037] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:38.775918 12248 raft_consensus.cc:504] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.776588 12248 leader_election.cc:288] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [CANDIDATE]: Term 1 election: Requested vote from peers 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397), be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075)
I20260808 15:58:38.777067 12177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87e733912d64949b3db1912fd6f2653" candidate_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7792734edf4738a59df72d2bf1a77e"
I20260808 15:58:38.777067 11908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87e733912d64949b3db1912fd6f2653" candidate_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
I20260808 15:58:38.777283 11908 raft_consensus.cc:3037] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:38.777271 12177 raft_consensus.cc:3037] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:58:38.780522 11908 raft_consensus.cc:2454] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4fb09f6ed564c9c82b663e3d3691bf7 in term 1.
I20260808 15:58:38.780519 12177 raft_consensus.cc:2454] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4fb09f6ed564c9c82b663e3d3691bf7 in term 1.
I20260808 15:58:38.780898 11978 leader_election.cc:302] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e; no voters:
I20260808 15:58:38.781237 12248 raft_consensus.cc:2781] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:58:38.781688 12248 raft_consensus.cc:686] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 LEADER]: Becoming Leader. State: Replica: a4fb09f6ed564c9c82b663e3d3691bf7, State: Running, Role: LEADER
I20260808 15:58:38.782171 12248 consensus_queue.cc:237] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } }
I20260808 15:58:38.786453 11763 catalog_manager.cc:5495] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } health_report { overall_health: HEALTHY } } }
W20260808 15:58:38.788599 11955 tablet.cc:2369] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:58:38.800572 12090 tablet.cc:2369] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:38.817694 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.812195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.27.254.2:40895)
0808 15:58:38.812435 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.812439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.812457 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:58:38.812634 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.812638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.812714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.812842 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:58:38.813164 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.813959 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.815509 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.815916 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.816008 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.816565 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:58:38.817087 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:58:38.817195 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.817295 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.817361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:58:38.825923 11908 raft_consensus.cc:1261] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Refusing update from remote peer a4fb09f6ed564c9c82b663e3d3691bf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:38.825923 12177 raft_consensus.cc:1261] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Refusing update from remote peer a4fb09f6ed564c9c82b663e3d3691bf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:58:38.827096 12248 consensus_queue.cc:1035] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:58:38.827430 12253 consensus_queue.cc:1035] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:58:38.828703 12224 tablet.cc:2369] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:58:38.838703 12256 mvcc.cc:205] Tried to move back new op lower bound from 7316294528299544576 to 7316294528135352320. Current Snapshot: MvccSnapshot[applied={T|T < 7316294528299544576}]
I20260808 15:58:38.839730 12258 mvcc.cc:205] Tried to move back new op lower bound from 7316294528299544576 to 7316294528135352320. Current Snapshot: MvccSnapshot[applied={T|T < 7316294528299544576}]
I20260808 15:58:38.857189 11763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:38.857702 11763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:58:38.861542 11763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Sending DeleteTablet for 3 replicas of tablet a87e733912d64949b3db1912fd6f2653
I20260808 15:58:38.862592 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet a87e733912d64949b3db1912fd6f2653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:38 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:42798
I20260808 15:58:38.862737 12157 tablet_service.cc:1514] Processing DeleteTablet for tablet a87e733912d64949b3db1912fd6f2653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:38 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37920
I20260808 15:58:38.863006 12023 tablet_service.cc:1514] Processing DeleteTablet for tablet a87e733912d64949b3db1912fd6f2653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:58:38 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:52552
I20260808 15:58:38.863941 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 11825
I20260808 15:58:38.891575 12268 tablet_replica.cc:330] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: stopping tablet replica
I20260808 15:58:38.892215 12268 raft_consensus.cc:2229] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:58:38.893116 12268 raft_consensus.cc:2258] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:38.895674 12266 tablet_replica.cc:330] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: stopping tablet replica
I20260808 15:58:38.896186 12266 raft_consensus.cc:2229] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:38.896826 12266 raft_consensus.cc:2258] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:38.904776 12268 ts_tablet_manager.cc:1905] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:58:38 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:58:38.910583 12267 tablet_replica.cc:330] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: stopping tablet replica
I20260808 15:58:38.911305 12267 raft_consensus.cc:2229] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:58:38.911831 12267 raft_consensus.cc:2258] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:58:38.912081 12266 ts_tablet_manager.cc:1905] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:38.913964 12267 ts_tablet_manager.cc:1905] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:58:38.917184 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:38.865487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.27.254.1:44397)
0808 15:58:38.865709 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:38.865718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:38.888456 (+ 22738us) server_negotiation.cc:407] Connection header received
0808 15:58:38.889040 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:38.889049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:38.889177 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:38.889291 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:38.897327 (+ 8036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.898929 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.901979 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:38.902617 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:38.902749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:38.908815 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:38.908831 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:38.908851 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:38.908897 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:38.912967 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.914129 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.914140 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.914145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.914355 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:38.914878 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:38.914882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:38.914886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:38.915406 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:38.915724 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:38.916434 (+ 710us) server_negotiation.cc:299] Negotiation successful
0808 15:58:38.916741 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:58:38.918883 11949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:38.921463 12267 ts_tablet_manager.cc:1918] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:38.921470 12268 ts_tablet_manager.cc:1918] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:58:38.921636 12267 log.cc:1196] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/a87e733912d64949b3db1912fd6f2653
I20260808 15:58:38.921655 12268 log.cc:1196] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/a87e733912d64949b3db1912fd6f2653
I20260808 15:58:38.922183 12267 ts_tablet_manager.cc:1939] T a87e733912d64949b3db1912fd6f2653 P be7792734edf4738a59df72d2bf1a77e: Deleting consensus metadata
I20260808 15:58:38.922191 12268 ts_tablet_manager.cc:1939] T a87e733912d64949b3db1912fd6f2653 P a4fb09f6ed564c9c82b663e3d3691bf7: Deleting consensus metadata
I20260808 15:58:38.924013 11749 catalog_manager.cc:4841] TS be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): tablet a87e733912d64949b3db1912fd6f2653 (table test-table [id=a676286cefe24bfe81c19ceb31933133]) successfully deleted
I20260808 15:58:38.924417 11749 catalog_manager.cc:4841] TS a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): tablet a87e733912d64949b3db1912fd6f2653 (table test-table [id=a676286cefe24bfe81c19ceb31933133]) successfully deleted
I20260808 15:58:39.015026 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 11825
I20260808 15:58:39.038036 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 11960
W20260808 15:58:39.037943 11747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:44397 (error 108)
I20260808 15:58:39.040400 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.039775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42816)
0808 15:58:39.040054 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.040194 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260808 15:58:39.040666 11747 catalog_manager.cc:4584] TS 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a87e733912d64949b3db1912fd6f2653: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Aug 08 15:58:39 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:58:39.054955 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.039200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.27.254.2:40895)
0808 15:58:39.039382 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:39.039386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:39.039404 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:58:39.039577 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:39.039581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:39.039642 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:39.039779 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:39.041497 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:39.042323 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:39.043699 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:39.044129 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:39.044280 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:39.048646 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:39.048656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:39.048668 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:39.048706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:39.051838 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:39.052594 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:39.052605 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:39.052608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:39.052762 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:39.053159 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:39.053165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:39.053168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:39.053623 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:39.053959 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:39.054243 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 15:58:39.054461 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:58:39.055665 12084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:39.094344 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.093888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42832)
0808 15:58:39.094092 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.094190 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 15:58:39.095350 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.094997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42848)
0808 15:58:39.095141 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.095234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 15:58:39.156095 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.155493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42858)
0808 15:58:39.155772 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.155886 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260808 15:58:39.157286 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.156909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42862)
0808 15:58:39.157073 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.157153 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:58:39.171305 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 11960
I20260808 15:58:39.190294 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 12094
Aug 08 15:58:39 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:58:39.206034 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.191080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.27.254.3:37075)
0808 15:58:39.191267 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:58:39.191275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:58:39.191332 (+ 57us) server_negotiation.cc:407] Connection header received
0808 15:58:39.191550 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:58:39.191561 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:58:39.191626 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:58:39.191733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:58:39.193011 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:39.193808 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:39.195078 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:58:39.195470 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:58:39.195612 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:58:39.200468 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:58:39.200482 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:58:39.200498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:58:39.200534 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:58:39.203315 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:39.203981 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:39.203988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:39.203990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:39.204118 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:58:39.204493 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:58:39.204499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:58:39.204502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:58:39.204856 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:58:39.205141 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 15:58:39.205499 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 15:58:39.205723 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:58:39.206782 12217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:39.238018 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.237556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42866)
0808 15:58:39.237782 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.237866 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:58:39.239046 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.238657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42882)
0808 15:58:39.238864 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.238931 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 15:58:39.295665 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 12094
I20260808 15:58:39.313740 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 11731
I20260808 15:58:39.314888 11793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:58:39.424525 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:58:39.423899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:42892)
0808 15:58:39.424088 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 15:58:39.424219 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 15:58:39.424841 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 11731
I20260808 15:58:59.446420 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33817
--webserver_interface=127.27.254.62
--webserver_port=39447
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:58:59.781687 12275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:58:59.781973 12275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:58:59.782061 12275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:58:59.789193 12275 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:58:59.789288 12275 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:58:59.789326 12275 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:58:59.789373 12275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:58:59.789487 12275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:58:59.789562 12275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:58:59.789606 12275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:58:59.789657 12275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:58:59.799635 12275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:58:59.800839 12275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:58:59.802160 12275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:58:59.824368 12281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:59.824337 12284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:58:59.824430 12282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:58:59.825258 12275 server_base.cc:1034] running on GCE node
I20260808 15:58:59.826196 12275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:58:59.827767 12275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:58:59.828998 12275 hybrid_clock.cc:648] HybridClock initialized: now 1786204739828926 us; error 40 us; skew 500 ppm
I20260808 15:58:59.829381 12275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:58:59 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204739, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:58:59.837831 12275 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:58:59.839498 12275 webserver.cc:460] Webserver started at http://127.27.254.62:39447/ using document root <none> and password file <none>
I20260808 15:58:59.840277 12275 fs_manager.cc:359] Metadata directory not provided
I20260808 15:58:59.840430 12275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:58:59.846503 12275 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:58:59.849484 12291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:58:59.850921 12275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:58:59.851117 12275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "bc7a9df785204bd4b5be5dac922ddd09"
format_stamp: "Formatted at 2026-08-08 15:58:36 on dist-test-slave-srgv"
server_key: "4be02cc4c9e62e562cca4dc9d4242db5"
server_key_iv: "92abde8d060e32bc0f94cf50822b435e"
server_key_version: "encryptionkey@0"
I20260808 15:58:59.852171 12275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:58:59.871709 12275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:58:59.875874 12275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:58:59.876186 12275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:58:59.895474 12275 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33817
I20260808 15:58:59.895509 12342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33817 every 8 connection(s)
I20260808 15:58:59.897030 12275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:58:59.901160 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12275
I20260808 15:58:59.902040 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:44397
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:33817
--webserver_port=41403
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:58:59.904032 12343 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:58:59.907316 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap starting.
I20260808 15:58:59.926745 12343 log.cc:824] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Log is configured to *not* fsync() on all Append() calls
I20260808 15:58:59.938745 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:58:59.939363 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap complete.
I20260808 15:58:59.953209 12343 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:59.954787 12343 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7a9df785204bd4b5be5dac922ddd09, State: Initialized, Role: FOLLOWER
I20260808 15:58:59.955685 12343 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:59.956065 12343 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:58:59.956226 12343 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:58:59.956457 12343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:58:59.962476 12343 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:59.963107 12343 leader_election.cc:302] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7a9df785204bd4b5be5dac922ddd09; no voters:
I20260808 15:58:59.963740 12343 leader_election.cc:288] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:58:59.963889 12347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:58:59.965091 12347 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 LEADER]: Becoming Leader. State: Replica: bc7a9df785204bd4b5be5dac922ddd09, State: Running, Role: LEADER
I20260808 15:58:59.965693 12347 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:58:59.965914 12343 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:58:59.967971 12349 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7a9df785204bd4b5be5dac922ddd09. Latest consensus state: current_term: 2 leader_uuid: "bc7a9df785204bd4b5be5dac922ddd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } } }
I20260808 15:58:59.968427 12349 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:59.968950 12348 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bc7a9df785204bd4b5be5dac922ddd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } } }
I20260808 15:58:59.969280 12348 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: This master's current role is: LEADER
I20260808 15:58:59.980262 12355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:58:59.988169 12355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:58:59.989604 12355 catalog_manager.cc:1259] Loaded cluster ID: 1b05373bf6c048ed95da5efc2b794c78
I20260808 15:58:59.989717 12355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:58:59.993795 12355 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:58:59.995323 12355 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Loaded TSK: 0
I20260808 15:59:00.011677 12365 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:59:00.028216 12355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Generated new TSK 1
I20260808 15:59:00.028651 12355 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:59:00.275967 12345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:00.276317 12345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:00.276482 12345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:00.284229 12345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:00.284392 12345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:00.284578 12345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:00.294853 12345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:44397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=41403
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:00.296231 12345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:00.297645 12345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:00.312625 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:00.312659 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:00.313588 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:00.314213 12345 server_base.cc:1034] running on GCE node
I20260808 15:59:00.314847 12345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:00.315955 12345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:00.317180 12345 hybrid_clock.cc:648] HybridClock initialized: now 1786204740317104 us; error 68 us; skew 500 ppm
I20260808 15:59:00.317570 12345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:00 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204740, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:00.320850 12345 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:59:00.322471 12345 webserver.cc:460] Webserver started at http://127.27.254.1:41403/ using document root <none> and password file <none>
I20260808 15:59:00.323191 12345 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:00.323362 12345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:00.328896 12345 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 15:59:00.332098 12381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:00.333669 12345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:59:00.333846 12345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "359ec443e3ce9f8efc299d32fc1cf884"
server_key_iv: "0221a19532f889d2388e07ce8f488f3b"
server_key_version: "encryptionkey@0"
I20260808 15:59:00.334796 12345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:00.360765 12345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:00.364817 12345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:00.365109 12345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:00.366243 12345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:00.368057 12388 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260808 15:59:00.373382 12388 ts_tablet_manager.cc:1852] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 15:59:00.374440 12388 ts_tablet_manager.cc:1905] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:00.381711 12388 ts_tablet_manager.cc:1918] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:00.381919 12388 log.cc:1196] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/a87e733912d64949b3db1912fd6f2653
I20260808 15:59:00.382418 12388 ts_tablet_manager.cc:1939] T a87e733912d64949b3db1912fd6f2653 P 6e19dc8640354a5a86a3c8a3b7627a67: Deleting consensus metadata
I20260808 15:59:00.382813 12345 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 15:59:00.382951 12345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260808 15:59:00.383092 12345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:00.383180 12345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:00.416558 12345 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:44397
I20260808 15:59:00.416604 12496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:44397 every 8 connection(s)
I20260808 15:59:00.418017 12345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:00.421926 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12345
I20260808 15:59:00.422679 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:40895
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:33817
--webserver_port=38827
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:59:00 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204740, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:00.442673 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:00.420996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:45269 (local address 127.27.254.62:33817)
0808 15:59:00.421733 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:00.421751 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:00.422390 (+ 639us) server_negotiation.cc:407] Connection header received
0808 15:59:00.424684 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:00.424696 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:00.425054 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:00.425450 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:00.427022 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.428243 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.429928 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.430587 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.430799 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:00.434853 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:00.434860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:00.434875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:00.434917 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:00.437880 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:00.438852 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:00.438858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:00.438863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:00.439033 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:00.439531 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:00.439535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:00.439536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:00.440089 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:00.440300 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:00.441290 (+ 990us) server_negotiation.cc:299] Negotiation successful
0808 15:59:00.441688 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":294,"threads_started":1}
I20260808 15:59:00.443171 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:00.421264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:45269)
0808 15:59:00.422200 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:00.422238 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:00.424124 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:00.425602 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:00.425613 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:00.426247 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:00.426785 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:00.426816 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.428417 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.428430 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:00.429773 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:00.429786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.430849 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.430857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:00.431113 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:00.431838 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:00.431866 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:00.434559 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:00.438057 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:00.438069 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:00.438088 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:00.438710 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:00.439180 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:00.439187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:00.439193 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:00.439420 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:00.440243 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:00.440254 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:00.440689 (+ 435us) client_negotiation.cc:769] Sending connection context
0808 15:59:00.441036 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 15:59:00.441662 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":347,"threads_started":1}
I20260808 15:59:00.446113 12498 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33817
I20260808 15:59:00.446424 12498 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:00.447113 12498 heartbeater.cc:511] Master 127.27.254.62:33817 requested a full tablet report, sending...
I20260808 15:59:00.449296 12308 ts_manager.cc:194] Registered new tserver with Master: 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397)
I20260808 15:59:00.450866 12308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:45269
W20260808 15:59:00.761065 12502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:00.761379 12502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:00.761579 12502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:00.769539 12502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:00.769721 12502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:00.769909 12502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:00.779456 12502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:40895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=38827
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:00.780745 12502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:00.782068 12502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:00.795387 12507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:00.795442 12510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:00.795409 12508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:00.796531 12502 server_base.cc:1034] running on GCE node
I20260808 15:59:00.797106 12502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:00.798080 12502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:00.799273 12502 hybrid_clock.cc:648] HybridClock initialized: now 1786204740799204 us; error 60 us; skew 500 ppm
I20260808 15:59:00.799594 12502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:00 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204740, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:00.802802 12502 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:59:00.804186 12502 webserver.cc:460] Webserver started at http://127.27.254.2:38827/ using document root <none> and password file <none>
I20260808 15:59:00.804787 12502 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:00.804911 12502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:00.810045 12502 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:00.812538 12517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:00.813911 12502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:00.814078 12502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "a4fb09f6ed564c9c82b663e3d3691bf7"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "4fdd6d3c52a5f76051a4d42b199c9df7"
server_key_iv: "999a77a6259b27a160c395b94d52eb0a"
server_key_version: "encryptionkey@0"
I20260808 15:59:00.815060 12502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:00.831389 12502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:00.835765 12502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:00.836081 12502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:00.837076 12502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:00.838560 12502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:00.838670 12502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:00.838797 12502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:00.838912 12502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:00.873579 12502 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:40895
I20260808 15:59:00.873661 12630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:40895 every 8 connection(s)
I20260808 15:59:00.875094 12502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:00.879653 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12502
I20260808 15:59:00.880497 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:37075
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:33817
--webserver_port=44773
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:59:00 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204740, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:00.899565 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:00.877784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:42687 (local address 127.27.254.62:33817)
0808 15:59:00.877965 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:00.877970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:00.879120 (+ 1150us) server_negotiation.cc:407] Connection header received
0808 15:59:00.882062 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:00.882067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:00.882154 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:00.882242 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:00.884366 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.885037 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.887648 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.888124 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.888246 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:00.892263 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:00.892272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:00.892281 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:00.892330 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:00.895937 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:00.896843 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:00.896859 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:00.896865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:00.897019 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:00.897535 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:00.897540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:00.897543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:00.897909 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:00.898063 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:00.898951 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 15:59:00.899193 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:59:00.900930 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:00.878268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:42687)
0808 15:59:00.878966 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:00.879006 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:00.881725 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:00.882542 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:00.882556 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:00.883341 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:00.884175 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:00.884203 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.885223 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.885235 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:00.887470 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:00.887488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:00.888389 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:00.888399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:00.888745 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:00.889489 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:00.889524 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:00.892028 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:00.896088 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:00.896096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:00.896111 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:00.896693 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:00.897149 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:00.897154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:00.897158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:00.897384 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:00.898067 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:00.898075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:00.898527 (+ 452us) client_negotiation.cc:769] Sending connection context
0808 15:59:00.898825 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 15:59:00.899426 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":270,"threads_started":1}
I20260808 15:59:00.902513 12631 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33817
I20260808 15:59:00.902841 12631 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:00.903836 12631 heartbeater.cc:511] Master 127.27.254.62:33817 requested a full tablet report, sending...
I20260808 15:59:00.905671 12308 ts_manager.cc:194] Registered new tserver with Master: a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
I20260808 15:59:00.906769 12308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:42687
W20260808 15:59:01.218647 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:01.218952 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:01.219117 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:01.226403 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:01.226549 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:01.226725 12634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:01.236426 12634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:37075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=44773
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:01.237735 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:01.239107 12634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:01.252214 12639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:01.252278 12640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:01.252203 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:01.253168 12634 server_base.cc:1034] running on GCE node
I20260808 15:59:01.253829 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:01.254776 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:01.255971 12634 hybrid_clock.cc:648] HybridClock initialized: now 1786204741255934 us; error 33 us; skew 500 ppm
I20260808 15:59:01.256330 12634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:01 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204741, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:01.259594 12634 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:59:01.260957 12634 webserver.cc:460] Webserver started at http://127.27.254.3:44773/ using document root <none> and password file <none>
I20260808 15:59:01.261683 12634 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:01.261801 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:01.266987 12634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:01.269591 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:01.270890 12634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:01.271063 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "be7792734edf4738a59df72d2bf1a77e"
format_stamp: "Formatted at 2026-08-08 15:58:38 on dist-test-slave-srgv"
server_key: "db40d32ff7e079122f6342b7a4669325"
server_key_iv: "d7a29ed399d0de356adf8e782dd183a4"
server_key_version: "encryptionkey@0"
I20260808 15:59:01.272006 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:01.294196 12634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:01.298278 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:01.298581 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:01.299629 12634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:01.300953 12634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:01.301072 12634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:01.301203 12634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:01.301294 12634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:01.333906 12634 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37075
I20260808 15:59:01.333993 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37075 every 8 connection(s)
I20260808 15:59:01.335340 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:01.340353 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12634
Aug 08 15:59:01 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204741, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:01.355222 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.341264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.27.254.62:33817)
0808 15:59:01.341839 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.341859 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.342388 (+ 529us) server_negotiation.cc:407] Connection header received
0808 15:59:01.342541 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.342547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.342609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.342722 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:01.344167 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.344977 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.346415 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.346911 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.347044 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.349634 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:01.349639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:01.349643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:01.349688 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:01.352477 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.353162 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.353167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.353173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.353295 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.353701 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.353706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.353710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.353993 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:01.354286 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.354619 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.354831 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":428,"thread_start_us":474,"threads_started":1}
I20260808 15:59:01.356251 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.337913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:36171 (local address 127.27.254.62:33817)
0808 15:59:01.338099 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.338103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.339351 (+ 1248us) server_negotiation.cc:407] Connection header received
0808 15:59:01.341314 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.341319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.341423 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.341542 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:01.343529 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.344253 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.345874 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.346299 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.346429 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.349919 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:01.349926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:01.349929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:01.349968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:01.352909 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.353652 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.353659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.353662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.353779 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.354261 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.354265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.354267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.354509 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:01.354653 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.355689 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.355920 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260808 15:59:01.357183 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.338422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:36171)
0808 15:59:01.339175 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.339212 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.341005 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.341997 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.342008 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.342764 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:01.343364 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.343381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.344432 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.344437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.345735 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.345748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.346543 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.346553 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.346786 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.347461 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:01.347520 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:01.349715 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:01.353047 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:01.353055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:01.353069 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:01.353542 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:01.353929 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:01.353936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:01.353939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:01.354127 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:01.354647 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:01.354655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:01.354972 (+ 317us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.355320 (+ 348us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.355913 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":345,"threads_started":1}
I20260808 15:59:01.358552 12763 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33817
I20260808 15:59:01.358798 12763 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:01.359421 12763 heartbeater.cc:511] Master 127.27.254.62:33817 requested a full tablet report, sending...
I20260808 15:59:01.360687 12308 ts_manager.cc:194] Registered new tserver with Master: be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075)
I20260808 15:59:01.361492 12308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:36171
I20260808 15:59:01.370229 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:59:01.376859 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.371102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.27.254.62:33817)
0808 15:59:01.371296 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.371303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.371774 (+ 471us) server_negotiation.cc:407] Connection header received
0808 15:59:01.371944 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.371948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.372086 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.372226 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:01.372599 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.373342 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.374972 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.375473 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.375564 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.375982 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:01.376271 (+ 289us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 15:59:01.376637 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49460: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 15:59:01.376983 12500 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49460: token expired
W20260808 15:59:01.377602 12230 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.254.62:33817)
I20260808 15:59:01.389746 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.378096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.27.254.62:33817)
0808 15:59:01.378238 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.378242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.378260 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:59:01.378384 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.378387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.378437 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.378508 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:01.379751 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.380494 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.382008 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.382395 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.382482 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.384939 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:01.384943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:01.384946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:01.384974 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:01.387336 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.387958 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.387962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.387964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.388068 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.388375 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.388378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.388379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.388753 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:01.388943 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.389207 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.389353 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:59:01.392803 12308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:59:01.399926 12308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:59:01.437754 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.411437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.27.254.3:37075)
0808 15:59:01.427143 (+ 15706us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.427156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.427217 (+ 61us) server_negotiation.cc:407] Connection header received
0808 15:59:01.427352 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.427364 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.427699 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.429270 (+ 1571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.429646 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.431826 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.434913 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.436254 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.436387 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.436603 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.437225 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.437395 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15448,"thread_start_us":16488,"threads_started":1}
I20260808 15:59:01.437974 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.411164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37075 (local address 127.0.0.1:37816)
0808 15:59:01.414283 (+ 3119us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.414343 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.414509 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.428298 (+ 13789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.428310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.428365 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.428977 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.428996 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.432035 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.432044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.434760 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.434776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.436767 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.436776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.436975 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.437015 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.437173 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.437445 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2893,"mutex_wait_us":58,"thread_start_us":2746,"threads_started":1}
I20260808 15:59:01.440593 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.412112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36300)
0808 15:59:01.432626 (+ 20514us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.432646 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.432889 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.433583 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.433588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.433613 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.434281 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.434293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.435828 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.435836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.437709 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.437727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.439808 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.439813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.439973 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.440006 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.440100 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.440218 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20315,"mutex_wait_us":1,"thread_start_us":20463,"threads_started":1}
I20260808 15:59:01.440622 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.412178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.27.254.1:44397)
0808 15:59:01.412674 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.412687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.432771 (+ 20084us) server_negotiation.cc:407] Connection header received
0808 15:59:01.433067 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.433074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.433331 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.433580 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.434465 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.435687 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.437912 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.439329 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.439423 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.439623 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.440171 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.440330 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":258,"threads_started":1}
I20260808 15:59:01.442081 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.414924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.27.254.2:40895)
0808 15:59:01.430237 (+ 15313us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.430247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.433066 (+ 2819us) server_negotiation.cc:407] Connection header received
0808 15:59:01.433223 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.433230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.433397 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.433602 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.435239 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.436706 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.439323 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.440757 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.440876 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.441052 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.441615 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.441774 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15145,"thread_start_us":15322,"threads_started":1}
I20260808 15:59:01.442062 12697 tablet_service.cc:1467] Processing CreateTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 (DEFAULT_TABLE table=test-table [id=f1a66a3f6a6f4c4e82ef946052b3927a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:01.442687 12432 tablet_service.cc:1467] Processing CreateTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 (DEFAULT_TABLE table=test-table [id=f1a66a3f6a6f4c4e82ef946052b3927a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:01.443444 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c90d2a6e1a4196be93b41e21b7a262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:01.443799 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c90d2a6e1a4196be93b41e21b7a262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:01.449817 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.414901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40895 (local address 127.0.0.1:40428)
0808 15:59:01.432904 (+ 18003us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.432927 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.433058 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.433859 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.433864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.433894 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.435081 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.435092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.436847 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.436853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.439141 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.439155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.441162 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.441168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.441366 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.441442 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.441527 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.441648 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17864,"mutex_wait_us":23,"thread_start_us":17343,"threads_started":1}
I20260808 15:59:01.451738 12779 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Bootstrap starting.
I20260808 15:59:01.453630 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 (DEFAULT_TABLE table=test-table [id=f1a66a3f6a6f4c4e82ef946052b3927a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:01.454303 12498 heartbeater.cc:503] Master 127.27.254.62:33817 was elected leader, sending a full tablet report...
I20260808 15:59:01.455032 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c90d2a6e1a4196be93b41e21b7a262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:01.455499 12779 tablet_bootstrap.cc:654] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:01.456902 12779 log.cc:824] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:01.460119 12780 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Bootstrap starting.
I20260808 15:59:01.461867 12779 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: No bootstrap required, opened a new log
I20260808 15:59:01.462132 12779 ts_tablet_manager.cc:1397] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260808 15:59:01.463233 12780 tablet_bootstrap.cc:654] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:01.464193 12780 log.cc:824] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:01.467110 12780 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: No bootstrap required, opened a new log
I20260808 15:59:01.467355 12780 ts_tablet_manager.cc:1397] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 15:59:01.472163 12783 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Bootstrap starting.
I20260808 15:59:01.472782 12779 raft_consensus.cc:348] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.473246 12779 raft_consensus.cc:374] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:01.473352 12779 raft_consensus.cc:729] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e19dc8640354a5a86a3c8a3b7627a67, State: Initialized, Role: FOLLOWER
I20260808 15:59:01.473999 12779 consensus_queue.cc:260] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.475831 12779 ts_tablet_manager.cc:1428] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260808 15:59:01.475862 12783 tablet_bootstrap.cc:654] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:01.475634 12780 raft_consensus.cc:348] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.476120 12780 raft_consensus.cc:374] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:01.476249 12780 raft_consensus.cc:729] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7792734edf4738a59df72d2bf1a77e, State: Initialized, Role: FOLLOWER
I20260808 15:59:01.477012 12780 consensus_queue.cc:260] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.477505 12783 log.cc:824] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:01.478241 12763 heartbeater.cc:503] Master 127.27.254.62:33817 was elected leader, sending a full tablet report...
I20260808 15:59:01.478876 12780 ts_tablet_manager.cc:1428] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 15:59:01.481523 12783 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: No bootstrap required, opened a new log
I20260808 15:59:01.481875 12783 ts_tablet_manager.cc:1397] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260808 15:59:01.488972 12783 raft_consensus.cc:348] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.489332 12783 raft_consensus.cc:374] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:01.489475 12783 raft_consensus.cc:729] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fb09f6ed564c9c82b663e3d3691bf7, State: Initialized, Role: FOLLOWER
I20260808 15:59:01.490087 12783 consensus_queue.cc:260] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.491175 12631 heartbeater.cc:503] Master 127.27.254.62:33817 was elected leader, sending a full tablet report...
I20260808 15:59:01.491782 12783 ts_tablet_manager.cc:1428] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
W20260808 15:59:01.587002 12764 tablet.cc:2369] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:59:01.627195 12632 tablet.cc:2369] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:59:01.670452 12499 tablet.cc:2369] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:01.718196 12787 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:01.718585 12787 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.718861 12785 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:01.719115 12785 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.720994 12785 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895), 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397)
I20260808 15:59:01.726670 12787 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075), 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397)
I20260808 15:59:01.734077 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.722504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:49945 (local address 127.27.254.2:40895)
0808 15:59:01.722747 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.722755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.722777 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:59:01.722871 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.722879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.722973 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.723081 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.724269 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.725895 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.729126 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.730942 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.731125 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.731240 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.733510 (+ 2270us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.733702 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:59:01.734537 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.721600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40895 (local address 127.27.254.3:49945)
0808 15:59:01.721962 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.721987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.722139 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.723740 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.723748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.723788 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.724066 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.724081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.726112 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.726121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.728788 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.728820 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.731388 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.731397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.732818 (+ 1421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.732875 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.732990 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.733189 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260808 15:59:01.736122 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c90d2a6e1a4196be93b41e21b7a262" candidate_uuid: "be7792734edf4738a59df72d2bf1a77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" is_pre_election: true
I20260808 15:59:01.736625 12586 raft_consensus.cc:2454] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be7792734edf4738a59df72d2bf1a77e in term 0.
I20260808 15:59:01.737454 12652 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e; no voters:
I20260808 15:59:01.738029 12785 raft_consensus.cc:2781] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:59:01.738193 12785 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:59:01.738302 12785 raft_consensus.cc:3037] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:01.741691 12785 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.742390 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.722401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.27.254.3:53617)
0808 15:59:01.734824 (+ 12423us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.734847 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.734983 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.735378 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.735386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.735424 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.735649 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.735663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.737603 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.737608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.739640 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.739654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.741426 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.741435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.741663 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.741710 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.741789 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.741982 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12285,"mutex_wait_us":52,"thread_start_us":13827,"threads_started":1}
I20260808 15:59:01.743216 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c90d2a6e1a4196be93b41e21b7a262" candidate_uuid: "be7792734edf4738a59df72d2bf1a77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7"
I20260808 15:59:01.743320 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.723361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:53617 (local address 127.27.254.1:44397)
0808 15:59:01.733060 (+ 9699us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.733070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.735003 (+ 1933us) server_negotiation.cc:407] Connection header received
0808 15:59:01.735102 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.735110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.735191 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.735806 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.736106 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.737393 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.739890 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.741184 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.742744 (+ 1560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.742806 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.742855 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.742979 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9489,"thread_start_us":9172,"threads_started":1}
I20260808 15:59:01.743450 12586 raft_consensus.cc:3037] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:01.743880 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.721469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:37689 (local address 127.27.254.1:44397)
0808 15:59:01.721735 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.721745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.735996 (+ 14251us) server_negotiation.cc:407] Connection header received
0808 15:59:01.736230 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.736238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.736330 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.736403 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.737233 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.738395 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.741019 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.742245 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.742379 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.742455 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.743528 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.743672 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:59:01.744035 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.736493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:35839 (local address 127.27.254.3:37075)
0808 15:59:01.736695 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.736701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.736718 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:59:01.736776 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.736782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.736858 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.736945 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:01.737871 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.739089 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.741589 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.742935 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.743049 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.743114 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.743654 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.743771 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":77}
I20260808 15:59:01.744326 12785 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [CANDIDATE]: Term 1 election: Requested vote from peers a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895), 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397)
I20260808 15:59:01.744724 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.721147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37075 (local address 127.27.254.2:35839)
0808 15:59:01.735058 (+ 13911us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.735081 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.735193 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.737230 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.737235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.737259 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.737637 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.737653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.739322 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.739327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.741432 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.741446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.743314 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.743320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.743497 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.743524 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.744188 (+ 664us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.744337 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13747,"thread_start_us":14866,"threads_started":1}
I20260808 15:59:01.744508 12452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c90d2a6e1a4196be93b41e21b7a262" candidate_uuid: "be7792734edf4738a59df72d2bf1a77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" is_pre_election: true
I20260808 15:59:01.745087 12452 raft_consensus.cc:2454] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be7792734edf4738a59df72d2bf1a77e in term 0.
I20260808 15:59:01.745355 12451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c90d2a6e1a4196be93b41e21b7a262" candidate_uuid: "be7792734edf4738a59df72d2bf1a77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
I20260808 15:59:01.745658 12451 raft_consensus.cc:3037] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:01.745699 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c90d2a6e1a4196be93b41e21b7a262" candidate_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7792734edf4738a59df72d2bf1a77e" is_pre_election: true
I20260808 15:59:01.746168 12718 raft_consensus.cc:2379] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4fb09f6ed564c9c82b663e3d3691bf7 in current term 1: Already voted for candidate be7792734edf4738a59df72d2bf1a77e in this term.
I20260808 15:59:01.746999 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.721205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.27.254.2:37689)
0808 15:59:01.735890 (+ 14685us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:01.735904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:01.736048 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:01.736593 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:01.736598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:01.736627 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:01.737051 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.737066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.738594 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.738600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.740835 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:01.740851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.743092 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.743102 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:01.743333 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.743368 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:59:01.745162 (+ 1794us) client_negotiation.cc:240] Negotiation successful
0808 15:59:01.745305 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14566,"spinlock_wait_cycles":22912,"thread_start_us":13514,"threads_started":1}
I20260808 15:59:01.747514 12586 raft_consensus.cc:2454] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be7792734edf4738a59df72d2bf1a77e in term 1.
I20260808 15:59:01.747560 12452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c90d2a6e1a4196be93b41e21b7a262" candidate_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" is_pre_election: true
I20260808 15:59:01.747925 12652 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e; no voters:
I20260808 15:59:01.748208 12785 raft_consensus.cc:2781] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:01.748430 12518 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4fb09f6ed564c9c82b663e3d3691bf7; no voters: 6e19dc8640354a5a86a3c8a3b7627a67, be7792734edf4738a59df72d2bf1a77e
I20260808 15:59:01.748729 12785 raft_consensus.cc:686] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 1 LEADER]: Becoming Leader. State: Replica: be7792734edf4738a59df72d2bf1a77e, State: Running, Role: LEADER
I20260808 15:59:01.748961 12451 raft_consensus.cc:2454] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be7792734edf4738a59df72d2bf1a77e in term 1.
I20260808 15:59:01.749140 12787 raft_consensus.cc:2726] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 15:59:01.749686 12785 consensus_queue.cc:237] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:01.753890 12307 catalog_manager.cc:5495] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e reported cstate change: term changed from 0 to 1, leader changed from <none> to be7792734edf4738a59df72d2bf1a77e (127.27.254.3). New cstate: current_term: 1 leader_uuid: "be7792734edf4738a59df72d2bf1a77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20260808 15:59:01.803817 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.798789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.27.254.3:37075)
0808 15:59:01.798975 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.798980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.799016 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:59:01.799201 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.799205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.799283 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.799405 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:01.799724 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.800450 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.801843 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.802220 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.802315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.802751 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:01.803227 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:59:01.803333 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.803421 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.803484 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:59:01.811645 12451 raft_consensus.cc:1261] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Refusing update from remote peer be7792734edf4738a59df72d2bf1a77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:01.811700 12586 raft_consensus.cc:1261] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Refusing update from remote peer be7792734edf4738a59df72d2bf1a77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:01.812613 12792 consensus_queue.cc:1035] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:01.812989 12785 consensus_queue.cc:1035] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:01.823812 12795 mvcc.cc:205] Tried to move back new op lower bound from 7316294622450102272 to 7316294622209957888. Current Snapshot: MvccSnapshot[applied={T|T < 7316294622450102272}]
I20260808 15:59:01.825011 12796 mvcc.cc:205] Tried to move back new op lower bound from 7316294622450102272 to 7316294622209957888. Current Snapshot: MvccSnapshot[applied={T|T < 7316294622450102272}]
I20260808 15:59:01.840617 12308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:01.841003 12308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:01.845507 12308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Sending DeleteTablet for 3 replicas of tablet 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:01.846669 12432 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:36300
I20260808 15:59:01.846683 12697 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:37816
I20260808 15:59:01.846900 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:40428
I20260808 15:59:01.847599 12805 tablet_replica.cc:330] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: stopping tablet replica
I20260808 15:59:01.847492 12564 tablet_service.cc:1514] Processing DeleteTablet for tablet 38c90d2a6e1a4196be93b41e21b7a262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:01 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:40428
I20260808 15:59:01.848095 12805 raft_consensus.cc:2229] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:59:01.848659 12805 raft_consensus.cc:2258] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:01.848773 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 12345
I20260808 15:59:01.850780 12805 ts_tablet_manager.cc:1905] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:01.858731 12805 ts_tablet_manager.cc:1918] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:01.859035 12805 log.cc:1196] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:01.859843 12805 ts_tablet_manager.cc:1939] T 38c90d2a6e1a4196be93b41e21b7a262 P be7792734edf4738a59df72d2bf1a77e: Deleting consensus metadata
I20260808 15:59:01.861912 12294 catalog_manager.cc:4841] TS be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): tablet 38c90d2a6e1a4196be93b41e21b7a262 (table test-table [id=f1a66a3f6a6f4c4e82ef946052b3927a]) successfully deleted
I20260808 15:59:01.862884 12807 tablet_replica.cc:330] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: stopping tablet replica
I20260808 15:59:01.863468 12807 raft_consensus.cc:2229] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:01.864116 12807 raft_consensus.cc:2258] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:01.866369 12807 ts_tablet_manager.cc:1905] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:01.868944 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:01.849851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.27.254.1:44397)
0808 15:59:01.850073 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:01.850082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:01.850318 (+ 236us) server_negotiation.cc:407] Connection header received
0808 15:59:01.850399 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:01.850406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:01.850483 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:01.850579 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:01.852674 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.853901 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.856015 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:01.856733 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:01.856869 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:01.860718 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:01.860736 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:01.860755 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:01.860803 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:01.864440 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.865399 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.865433 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.865437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.865671 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:01.866169 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:01.866174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:01.866177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:01.866666 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:01.867064 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:01.868287 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0808 15:59:01.868540 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:59:01.870121 12492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:01.874891 12807 ts_tablet_manager.cc:1918] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:01.875118 12807 log.cc:1196] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:01.876035 12807 ts_tablet_manager.cc:1939] T 38c90d2a6e1a4196be93b41e21b7a262 P a4fb09f6ed564c9c82b663e3d3691bf7: Deleting consensus metadata
I20260808 15:59:01.878547 12294 catalog_manager.cc:4841] TS a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): tablet 38c90d2a6e1a4196be93b41e21b7a262 (table test-table [id=f1a66a3f6a6f4c4e82ef946052b3927a]) successfully deleted
W20260808 15:59:01.879832 12294 catalog_manager.cc:4808] TS a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): delete failed for tablet 38c90d2a6e1a4196be93b41e21b7a262 because the tablet was not found. No further retry: Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:01.995922 12806 tablet_replica.cc:330] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: stopping tablet replica
I20260808 15:59:01.996423 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 12345
W20260808 15:59:02.020444 12292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:44397 (error 108)
I20260808 15:59:02.021303 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 12502
I20260808 15:59:02.023034 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.021777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36314)
0808 15:59:02.022685 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.022806 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":688,"threads_started":1}
W20260808 15:59:02.023317 12292 catalog_manager.cc:4584] TS 6e19dc8640354a5a86a3c8a3b7627a67 (127.27.254.1:44397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38c90d2a6e1a4196be93b41e21b7a262: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
I20260808 15:59:02.037192 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.022158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.27.254.2:40895)
0808 15:59:02.022343 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:02.022361 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:02.022778 (+ 417us) server_negotiation.cc:407] Connection header received
0808 15:59:02.022960 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:02.022969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:02.023055 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:02.023161 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:02.024727 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:02.025888 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:02.027140 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:02.027783 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:02.027912 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:02.030549 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:02.030565 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:02.030584 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:02.030634 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:02.034426 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:02.035086 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:02.035097 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:02.035101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:02.035265 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:02.035631 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:02.035636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:02.035638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:02.036082 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:02.036386 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:02.036556 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 15:59:02.036807 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:59:02.037927 12626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:02.077260 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.076711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36328)
0808 15:59:02.076954 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.077082 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 15:59:02.078502 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.078035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36342)
0808 15:59:02.078220 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.078322 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:59:02.131580 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 12502
I20260808 15:59:02.139140 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.138556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36344)
0808 15:59:02.138814 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.138938 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 15:59:02.140439 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.139969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36360)
0808 15:59:02.140156 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.140254 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 15:59:02.150482 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 12634
I20260808 15:59:02.166524 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.151505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.27.254.3:37075)
0808 15:59:02.151752 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:02.151759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:02.151781 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:59:02.151858 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:02.151864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:02.151946 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:02.152094 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:02.153318 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:02.154531 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:02.155926 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:02.156617 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:02.156756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:02.159416 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:02.159430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:02.159449 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:02.159488 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:02.163549 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:02.164239 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:02.164250 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:02.164256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:02.164440 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:02.164810 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:02.164815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:02.164818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:02.165309 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:02.165700 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:02.165891 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 15:59:02.166112 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:59:02.167498 12758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:02.221191 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.220610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36366)
0808 15:59:02.220882 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.220997 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 15:59:02.222425 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.221920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36370)
0808 15:59:02.222141 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.222228 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260808 15:59:02.289551 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 12634
I20260808 15:59:02.310737 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 12275
I20260808 15:59:02.311856 12338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:02.424700 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:02.424144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:44397 (local address 127.0.0.1:36386)
0808 15:59:02.424325 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:02.424503 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 15:59:02.424747 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 12275
I20260808 15:59:22.447124 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33817
--webserver_interface=127.27.254.62
--webserver_port=39447
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:22.764426 12814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:22.764786 12814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:22.764959 12814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:22.772012 12814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:59:22.772128 12814 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:59:22.772212 12814 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:59:22.772297 12814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:22.772389 12814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:22.772495 12814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:22.772578 12814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:22.772653 12814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:22.782114 12814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=39447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:22.783366 12814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:22.784663 12814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:22.796787 12819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:22.796739 12822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:22.796904 12820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:22.797511 12814 server_base.cc:1034] running on GCE node
I20260808 15:59:22.798267 12814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:22.799616 12814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:22.800844 12814 hybrid_clock.cc:648] HybridClock initialized: now 1786204762800777 us; error 56 us; skew 500 ppm
I20260808 15:59:22.801198 12814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:22 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204762, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:22.810892 12814 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:59:22.812359 12814 webserver.cc:460] Webserver started at http://127.27.254.62:39447/ using document root <none> and password file <none>
I20260808 15:59:22.813053 12814 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:22.813165 12814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:22.818341 12814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:22.820963 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:22.822253 12814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:22.822431 12814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "bc7a9df785204bd4b5be5dac922ddd09"
format_stamp: "Formatted at 2026-08-08 15:58:36 on dist-test-slave-srgv"
server_key: "4be02cc4c9e62e562cca4dc9d4242db5"
server_key_iv: "92abde8d060e32bc0f94cf50822b435e"
server_key_version: "encryptionkey@0"
I20260808 15:59:22.823385 12814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:22.838651 12814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:22.843052 12814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:22.843351 12814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:22.859645 12814 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33817
I20260808 15:59:22.859668 12881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33817 every 8 connection(s)
I20260808 15:59:22.861085 12814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:22.867718 12882 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:59:22.870743 12882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap starting.
I20260808 15:59:22.870873 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12814
I20260808 15:59:22.871639 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:44397
--local_ip_for_outbound_sockets=127.27.254.1
--tserver_master_addrs=127.27.254.62:33817
--webserver_port=41403
--webserver_interface=127.27.254.1
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:59:22.890355 12882 log.cc:824] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:22.905701 12882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:59:22.906229 12882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Bootstrap complete.
I20260808 15:59:22.915783 12882 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:59:22.916814 12882 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7a9df785204bd4b5be5dac922ddd09, State: Initialized, Role: FOLLOWER
I20260808 15:59:22.917560 12882 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:59:22.917793 12882 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:22.917902 12882 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:22.918077 12882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 2 FOLLOWER]: Advancing to term 3
I20260808 15:59:22.921852 12882 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:59:22.922329 12882 leader_election.cc:302] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7a9df785204bd4b5be5dac922ddd09; no voters:
I20260808 15:59:22.922951 12882 leader_election.cc:288] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 15:59:22.923190 12886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 15:59:22.924357 12886 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [term 3 LEADER]: Becoming Leader. State: Replica: bc7a9df785204bd4b5be5dac922ddd09, State: Running, Role: LEADER
I20260808 15:59:22.925034 12882 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:22.924980 12886 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } }
I20260808 15:59:22.928499 12888 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7a9df785204bd4b5be5dac922ddd09. Latest consensus state: current_term: 3 leader_uuid: "bc7a9df785204bd4b5be5dac922ddd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } } }
I20260808 15:59:22.929566 12888 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:22.929028 12887 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bc7a9df785204bd4b5be5dac922ddd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a9df785204bd4b5be5dac922ddd09" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33817 } } }
I20260808 15:59:22.929988 12887 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:22.938584 12896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:22.946856 12896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:22.948268 12896 catalog_manager.cc:1259] Loaded cluster ID: 1b05373bf6c048ed95da5efc2b794c78
I20260808 15:59:22.948370 12896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:22.952422 12896 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:22.953979 12896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Loaded TSK: 0
I20260808 15:59:22.954118 12896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc7a9df785204bd4b5be5dac922ddd09: Loaded TSK: 1
F20260808 15:59:22.955219 12896 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f785414c9ad google::LogMessage::Fail() at ??:0
@ 0x7f785414f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f785414c4de google::LogMessage::Flush() at ??:0
@ 0x7f785414da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7857eb07d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7857e9f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7869b7b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7869b7a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7869c062a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7869c060c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7869c505e9 std::function<>::operator()() at ??:0
@ 0x7f7869b7ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7869b7cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7869c04610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7869c04439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7856d98345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7856d9b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7856d9b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7856d76eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f785a0506db start_thread at ??:0
@ 0x7f7852fe661f clone at ??:0
W20260808 15:59:23.256026 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:23.256282 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:23.256374 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:23.263293 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:23.263398 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:23.263512 12884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:23.272754 12884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:44397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=41403
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:23.273908 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:23.275074 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:23.287761 12930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:23.287802 12931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:23.287837 12933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:23.288539 12884 server_base.cc:1034] running on GCE node
I20260808 15:59:23.289114 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:23.290097 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:23.291311 12884 hybrid_clock.cc:648] HybridClock initialized: now 1786204763291270 us; error 30 us; skew 500 ppm
I20260808 15:59:23.291605 12884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:23 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204763, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:23.294714 12884 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:59:23.296087 12884 webserver.cc:460] Webserver started at http://127.27.254.1:41403/ using document root <none> and password file <none>
I20260808 15:59:23.296701 12884 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:23.296799 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:23.302034 12884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:59:23.304721 12940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:23.306043 12884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:59:23.306209 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "6e19dc8640354a5a86a3c8a3b7627a67"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "359ec443e3ce9f8efc299d32fc1cf884"
server_key_iv: "0221a19532f889d2388e07ce8f488f3b"
server_key_version: "encryptionkey@0"
I20260808 15:59:23.307060 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:23.319802 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:23.323336 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:23.323582 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:23.324496 12884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:23.326227 12947 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 15:59:23.331772 12884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 15:59:23.331874 12884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260808 15:59:23.331998 12884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 15:59:23.335068 12884 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 15:59:23.335150 12884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260808 15:59:23.335512 12947 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Bootstrap starting.
I20260808 15:59:23.361152 12947 log.cc:824] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:23.376000 12884 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:44397
I20260808 15:59:23.376039 13055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:44397 every 8 connection(s)
I20260808 15:59:23.376658 12947 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:59:23.377202 12947 tablet_bootstrap.cc:492] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Bootstrap complete.
I20260808 15:59:23.377279 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:23.378875 12947 ts_tablet_manager.cc:1397] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Time spent bootstrapping tablet: real 0.044s user 0.029s sys 0.012s
I20260808 15:59:23.382237 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.380051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:43739)
0808 15:59:23.380972 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.381290 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":452,"threads_started":1}
W20260808 15:59:23.382942 13056 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:33817 (0 consecutive failures): Network error: Failed to ping master at 127.27.254.62:33817: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
I20260808 15:59:23.383711 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.383411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:52601)
0808 15:59:23.383547 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.383617 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 15:59:23.384492 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.384178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:40217)
0808 15:59:23.384303 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.384384 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 15:59:23.384752 13056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:59:23.385872 12947 raft_consensus.cc:348] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:23.386636 12947 raft_consensus.cc:729] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e19dc8640354a5a86a3c8a3b7627a67, State: Initialized, Role: FOLLOWER
I20260808 15:59:23.387089 12947 consensus_queue.cc:260] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:23.387956 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 12884
I20260808 15:59:23.388442 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.388144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:50409)
0808 15:59:23.388271 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.388345 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 15:59:23.388640 12947 ts_tablet_manager.cc:1428] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260808 15:59:23.388728 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:40895
--local_ip_for_outbound_sockets=127.27.254.2
--tserver_master_addrs=127.27.254.62:33817
--webserver_port=38827
--webserver_interface=127.27.254.2
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:59:23.389647 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.389285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:42335)
0808 15:59:23.389449 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.389524 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:59:23.706218 13060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:23.706480 13060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:23.706655 13060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:23.713984 13060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:23.714116 13060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:23.714298 13060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:23.723919 13060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:40895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=38827
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:23.725199 13060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:23.726516 13060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:23.739418 13065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:23.739501 13068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:23.739457 13066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:23.740427 13060 server_base.cc:1034] running on GCE node
I20260808 15:59:23.741037 13060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:23.742026 13060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:23.743255 13060 hybrid_clock.cc:648] HybridClock initialized: now 1786204763743203 us; error 41 us; skew 500 ppm
I20260808 15:59:23.743553 13060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:23 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204763, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:23.746567 13060 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:59:23.747895 13060 webserver.cc:460] Webserver started at http://127.27.254.2:38827/ using document root <none> and password file <none>
I20260808 15:59:23.748566 13060 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:23.748741 13060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:23.754031 13060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:23.756747 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:23.758064 13060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:59:23.758250 13060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "a4fb09f6ed564c9c82b663e3d3691bf7"
format_stamp: "Formatted at 2026-08-08 15:58:37 on dist-test-slave-srgv"
server_key: "4fdd6d3c52a5f76051a4d42b199c9df7"
server_key_iv: "999a77a6259b27a160c395b94d52eb0a"
server_key_version: "encryptionkey@0"
I20260808 15:59:23.759219 13060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:23.773101 13060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:23.776705 13060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:23.776960 13060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:23.777936 13060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:23.779160 13060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:23.779285 13060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:23.779397 13060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:23.779476 13060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:23.812618 13060 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:40895
I20260808 15:59:23.812675 13188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:40895 every 8 connection(s)
I20260808 15:59:23.814018 13060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:23.818539 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.816829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:32939)
0808 15:59:23.817546 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.817763 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":251,"threads_started":1}
W20260808 15:59:23.819199 13189 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:33817 (0 consecutive failures): Network error: Failed to ping master at 127.27.254.62:33817: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
I20260808 15:59:23.819964 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.819660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:45277)
0808 15:59:23.819798 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.819869 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 15:59:23.820695 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:23.820406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:59439)
0808 15:59:23.820529 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:23.820600 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 15:59:23.820941 13189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:59:23.822752 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13060
I20260808 15:59:23.823391 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:37075
--local_ip_for_outbound_sockets=127.27.254.3
--tserver_master_addrs=127.27.254.62:33817
--webserver_port=44773
--webserver_interface=127.27.254.3
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:24.144313 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:24.144582 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:24.144754 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:24.152000 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:24.152117 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:24.152303 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:24.161670 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:37075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=44773
--tserver_master_addrs=127.27.254.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:24.162948 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:24.164265 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:24.176771 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:24.176738 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:24.176849 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:24.177605 13192 server_base.cc:1034] running on GCE node
I20260808 15:59:24.178138 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:24.178977 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:24.180184 13192 hybrid_clock.cc:648] HybridClock initialized: now 1786204764180148 us; error 45 us; skew 500 ppm
I20260808 15:59:24.180500 13192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:24 dist-test-slave-srgv krb5kdc[11717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204764, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:24.183405 13192 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:59:24.184718 13192 webserver.cc:460] Webserver started at http://127.27.254.3:44773/ using document root <none> and password file <none>
I20260808 15:59:24.185366 13192 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:24.185524 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:24.190519 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 15:59:24.192960 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:24.194186 13192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 15:59:24.194350 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "be7792734edf4738a59df72d2bf1a77e"
format_stamp: "Formatted at 2026-08-08 15:58:38 on dist-test-slave-srgv"
server_key: "db40d32ff7e079122f6342b7a4669325"
server_key_iv: "d7a29ed399d0de356adf8e782dd183a4"
server_key_version: "encryptionkey@0"
I20260808 15:59:24.195309 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:24.222051 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:24.225669 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:24.225936 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:24.226821 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:24.228029 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:24.228139 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:24.228248 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:24.228350 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:24.259351 13192 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:37075
I20260808 15:59:24.259430 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:37075 every 8 connection(s)
I20260808 15:59:24.260761 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:24.265235 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.263566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:52265)
0808 15:59:24.264210 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.264442 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":252,"threads_started":1}
W20260808 15:59:24.265875 13322 heartbeater.cc:644] Failed to heartbeat to 127.27.254.62:33817 (0 consecutive failures): Network error: Failed to ping master at 127.27.254.62:33817: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
I20260808 15:59:24.266608 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.266302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:53169)
0808 15:59:24.266459 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.266520 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 15:59:24.266781 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13192
I20260808 15:59:24.267468 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.267164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:52619)
0808 15:59:24.267302 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.267372 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:59:24.267721 13322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:59:24.390998 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.390324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:57419)
0808 15:59:24.390710 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.390815 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":192,"threads_started":1}
I20260808 15:59:24.747380 13327 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:24.747731 13327 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:24.749451 13327 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075), a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
Aug 08 15:59:24 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204763, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
Aug 08 15:59:24 dist-test-slave-srgv krb5kdc[11717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204763, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:59:24.774380 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.749731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:37075 (local address 127.27.254.1:37557)
0808 15:59:24.750910 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.750935 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:24.751550 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:24.756072 (+ 4522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:24.756087 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:24.757043 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:24.757921 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:24.757953 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.760271 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.760275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:24.762115 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:24.762125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.763418 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.763424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:24.763707 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:24.764407 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:24.764434 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:24.767161 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:24.770902 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:24.770912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:24.770925 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:24.771495 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:24.771987 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:24.771992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:24.771996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:24.772221 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:24.772936 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:24.772944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:24.773314 (+ 370us) client_negotiation.cc:769] Sending connection context
0808 15:59:24.773616 (+ 302us) client_negotiation.cc:240] Negotiation successful
0808 15:59:24.773994 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"spinlock_wait_cycles":80640}
I20260808 15:59:24.774586 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.749987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:37557 (local address 127.27.254.3:37075)
0808 15:59:24.750803 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:24.750835 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:24.751038 (+ 203us) server_negotiation.cc:407] Connection header received
0808 15:59:24.754761 (+ 3723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:24.754793 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:24.755481 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:24.755902 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:24.758159 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.760077 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.762261 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.763220 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.763412 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:24.767454 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:24.767464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:24.767484 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:24.767532 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:24.770761 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:24.771651 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:24.771661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:24.771670 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:24.771868 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:24.772328 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:24.772331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:24.772333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:24.772795 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:24.772930 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:24.773883 (+ 953us) server_negotiation.cc:299] Negotiation successful
0808 15:59:24.774266 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":460,"threads_started":1}
I20260808 15:59:24.776489 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.750150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:40895 (local address 127.27.254.1:53917)
0808 15:59:24.751122 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.751139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:24.751552 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:24.760969 (+ 9417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:24.760974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:24.761508 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:24.761866 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:24.761877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.763937 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.763943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:24.765341 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:24.765355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.766471 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.766477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:24.766645 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:24.767215 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:24.767252 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:24.769768 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:24.773489 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:24.773493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:24.773495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:24.774061 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:24.774829 (+ 768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:24.774836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:24.774839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:24.774965 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:24.775615 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:24.775619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:24.775749 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 15:59:24.775877 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 15:59:24.776132 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":697,"threads_started":1}
I20260808 15:59:24.777045 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.750766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:53917 (local address 127.27.254.2:40895)
0808 15:59:24.758310 (+ 7544us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:24.758325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:24.758364 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:59:24.760367 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:24.760375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:24.760706 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:24.760944 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:24.762053 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.763804 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.765500 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:24.766269 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:24.766463 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:24.770009 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:24.770016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:24.770027 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:24.770060 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:24.773327 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:24.774197 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:24.774204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:24.774210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:24.774398 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:24.775072 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:24.775077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:24.775080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:24.775497 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:24.775671 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:24.776109 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 15:59:24.776452 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7251,"thread_start_us":7175,"threads_started":1}
W20260808 15:59:24.777293 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
W20260808 15:59:24.779186 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:24.779376 12943 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e19dc8640354a5a86a3c8a3b7627a67; no voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e
I20260808 15:59:24.779767 13327 raft_consensus.cc:2726] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:59:24.822194 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:24.821372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:42747)
0808 15:59:24.821817 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:24.821954 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":200,"threads_started":1}
I20260808 15:59:25.269032 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:25.268207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:60529)
0808 15:59:25.268654 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:25.268808 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":220,"threads_started":1}
I20260808 15:59:25.392504 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:25.391796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:50471)
0808 15:59:25.392256 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:25.392347 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":269,"threads_started":1}
I20260808 15:59:25.823680 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:25.822990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:59915)
0808 15:59:25.823464 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:25.823542 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":277,"threads_started":1}
I20260808 15:59:26.270617 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:26.269936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:58721)
0808 15:59:26.270370 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:26.270455 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260808 15:59:26.394171 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:26.393366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:52489)
0808 15:59:26.393915 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:26.393990 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":240,"threads_started":1}
I20260808 15:59:26.751896 13337 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:26.752102 13337 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:26.752813 13337 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075), a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
W20260808 15:59:26.753785 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
W20260808 15:59:26.754051 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:26.754151 12943 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e19dc8640354a5a86a3c8a3b7627a67; no voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e
I20260808 15:59:26.754405 13337 raft_consensus.cc:2726] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:59:26.825235 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:26.824580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:55061)
0808 15:59:26.825004 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:26.825084 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":218,"threads_started":1}
I20260808 15:59:27.272049 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:27.271357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:54045)
0808 15:59:27.271831 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:27.271903 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
I20260808 15:59:27.395644 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:27.395008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:58477)
0808 15:59:27.395412 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:27.395492 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260808 15:59:27.826733 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:27.826070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:45491)
0808 15:59:27.826473 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:27.826561 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":212,"threads_started":1}
I20260808 15:59:28.273689 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:28.272892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:57079)
0808 15:59:28.273342 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:28.273496 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260808 15:59:28.397468 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:28.396701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:58443)
0808 15:59:28.397154 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:28.397235 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
I20260808 15:59:28.828187 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:28.827521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:33663)
0808 15:59:28.827954 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:28.828038 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260808 15:59:29.275367 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:29.274535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:46183)
0808 15:59:29.275054 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:29.275145 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260808 15:59:29.399029 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:29.398312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:40403)
0808 15:59:29.398747 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:29.398825 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260808 15:59:29.638949 13347 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:29.639130 13347 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:29.639731 13347 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075), a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
W20260808 15:59:29.640723 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
W20260808 15:59:29.640957 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:29.641086 12943 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e19dc8640354a5a86a3c8a3b7627a67; no voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e
I20260808 15:59:29.641352 13347 raft_consensus.cc:2726] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:59:29.829779 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:29.829023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:45535)
0808 15:59:29.829507 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:29.829585 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260808 15:59:30.276896 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:30.276204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:38731)
0808 15:59:30.276639 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:30.276717 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260808 15:59:30.400599 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:30.399909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:58529)
0808 15:59:30.400331 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:30.400410 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":220,"threads_started":1}
I20260808 15:59:30.831564 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:30.830657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:34697)
0808 15:59:30.831273 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:30.831358 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":267,"threads_started":1}
I20260808 15:59:31.214712 13353 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:31.214903 13353 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:31.215492 13353 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075), a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
W20260808 15:59:31.216399 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
W20260808 15:59:31.216586 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:31.216658 12943 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e19dc8640354a5a86a3c8a3b7627a67; no voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e
I20260808 15:59:31.216955 13353 raft_consensus.cc:2726] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:59:31.278636 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:31.277822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:48931)
0808 15:59:31.278333 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:31.278451 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":244,"threads_started":1}
I20260808 15:59:31.402213 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:31.401380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:48853)
0808 15:59:31.401918 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:31.402003 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":250,"threads_started":1}
I20260808 15:59:31.833101 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:31.832434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:60131)
0808 15:59:31.832853 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:31.832929 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260808 15:59:32.280249 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:32.279483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:56233)
0808 15:59:32.279987 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:32.280063 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
I20260808 15:59:32.403673 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:32.402989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:40191)
0808 15:59:32.403426 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:32.403506 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260808 15:59:32.834918 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:32.834031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:34627)
0808 15:59:32.834694 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:32.834773 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":428,"threads_started":1}
I20260808 15:59:33.281864 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:33.281144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:35435)
0808 15:59:33.281641 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:33.281724 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
I20260808 15:59:33.405476 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:33.404657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:43043)
0808 15:59:33.405164 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:33.405264 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":282,"threads_started":1}
I20260808 15:59:33.414366 13362 raft_consensus.cc:482] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:33.414603 13362 raft_consensus.cc:504] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7792734edf4738a59df72d2bf1a77e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 37075 } } peers { permanent_uuid: "a4fb09f6ed564c9c82b663e3d3691bf7" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 40895 } } peers { permanent_uuid: "6e19dc8640354a5a86a3c8a3b7627a67" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 44397 } }
I20260808 15:59:33.415293 13362 leader_election.cc:288] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075), a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895)
W20260808 15:59:33.416278 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be7792734edf4738a59df72d2bf1a77e (127.27.254.3:37075): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
W20260808 15:59:33.416522 12943 leader_election.cc:341] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4fb09f6ed564c9c82b663e3d3691bf7 (127.27.254.2:40895): Not found: Tablet not found: 38c90d2a6e1a4196be93b41e21b7a262
I20260808 15:59:33.416666 12943 leader_election.cc:302] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e19dc8640354a5a86a3c8a3b7627a67; no voters: a4fb09f6ed564c9c82b663e3d3691bf7, be7792734edf4738a59df72d2bf1a77e
I20260808 15:59:33.416913 13362 raft_consensus.cc:2726] T 38c90d2a6e1a4196be93b41e21b7a262 P 6e19dc8640354a5a86a3c8a3b7627a67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:59:33.836556 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:33.835868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:48135)
0808 15:59:33.836308 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:33.836385 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":224,"threads_started":1}
I20260808 15:59:34.283286 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:34.282658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:33731)
0808 15:59:34.283034 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:34.283105 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":207,"threads_started":1}
I20260808 15:59:34.406946 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:34.406265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:52099)
0808 15:59:34.406701 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:34.406778 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260808 15:59:34.838264 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:34.837557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:38239)
0808 15:59:34.838019 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:34.838099 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":215,"threads_started":1}
I20260808 15:59:35.284935 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:35.284216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:49175)
0808 15:59:35.284691 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:35.284779 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":233,"threads_started":1}
I20260808 15:59:35.408372 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:35.407689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:43003)
0808 15:59:35.408117 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:35.408191 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":247,"threads_started":1}
I20260808 15:59:35.839785 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:35.839058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:49171)
0808 15:59:35.839506 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:35.839587 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":256,"threads_started":1}
I20260808 15:59:36.286550 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:36.285897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:50925)
0808 15:59:36.286314 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:36.286388 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":225,"threads_started":1}
I20260808 15:59:36.409897 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:36.409149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:49771)
0808 15:59:36.409641 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:36.409724 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":295,"threads_started":1}
I20260808 15:59:36.841398 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:36.840716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:38787)
0808 15:59:36.841138 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:36.841244 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260808 15:59:37.288046 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:37.287381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:35257)
0808 15:59:37.287817 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:37.287897 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260808 15:59:37.411481 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:37.410702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:38631)
0808 15:59:37.411161 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:37.411240 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260808 15:59:37.843108 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:37.842343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:48333)
0808 15:59:37.842834 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:37.842922 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260808 15:59:38.289719 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:38.288958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:44103)
0808 15:59:38.289348 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:38.289481 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":197,"threads_started":1}
I20260808 15:59:38.413136 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:38.412450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.1:47737)
0808 15:59:38.412875 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:38.412978 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260808 15:59:38.844697 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:38.844059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.2:35817)
0808 15:59:38.844459 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:38.844543 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260808 15:59:39.278618 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 12884
I20260808 15:59:39.291460 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:39.290769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33817 (local address 127.27.254.3:34247)
0808 15:59:39.291199 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:39.291302 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.62:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260808 15:59:39.298079 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:39.279634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.27.254.1:44397)
0808 15:59:39.280328 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:39.280335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:39.280359 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:59:39.280419 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:39.280425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:39.280650 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:39.280795 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:39.282431 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:39.283364 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:39.284594 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:39.285179 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:39.285329 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:39.292575 (+ 7246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:39.292587 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:39.292601 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:39.292634 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:39.295328 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:39.296009 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:39.296016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:39.296018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:39.296141 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:39.296520 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:39.296525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:39.296528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:39.296875 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:39.297198 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:39.297496 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 15:59:39.297777 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":409,"threads_started":1}
I20260808 15:59:39.299422 13051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:39.385160 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 12884
I20260808 15:59:39.403650 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13060
I20260808 15:59:39.416944 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:39.404430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.27.254.2:40895)
0808 15:59:39.404977 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:39.404981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:39.404996 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:59:39.405046 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:39.405049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:39.405105 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:39.405227 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:39.406717 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:39.407506 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:39.408746 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:39.409162 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:39.409313 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:39.411816 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:39.411821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:39.411824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:39.411854 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:39.414337 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:39.415033 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:39.415046 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:39.415049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:39.415180 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:39.415544 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:39.415550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:39.415553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:39.415812 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:39.416084 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:39.416373 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:59:39.416578 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
I20260808 15:59:39.417672 13184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:39.497192 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13060
I20260808 15:59:39.513888 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13192
I20260808 15:59:39.526165 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:39.514709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.27.254.3:37075)
0808 15:59:39.515271 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:39.515276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:39.515293 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:59:39.515347 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:39.515350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:39.515415 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:39.515514 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:39.516739 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:39.517502 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:39.518653 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:39.519055 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:39.519147 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:39.521622 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:39.521627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:39.521630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:39.521660 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:39.523863 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:39.524434 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:39.524438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:39.524440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:39.524557 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:39.524876 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:39.524879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:39.524881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:39.525134 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:39.525332 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:39.525724 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 15:59:39.525877 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
I20260808 15:59:39.526865 13317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:39.606405 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13192
2026-08-08T15:59:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64446 ms)
[----------] 30 tests from SecurityITest (294393 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:59:39 dist-test-slave-srgv krb5kdc[13388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:59:39 dist-test-slave-srgv krb5kdc[13388](info): set up 2 sockets
Aug 08 15:59:39 dist-test-slave-srgv krb5kdc[13388](info): commencing operation
krb5kdc: starting...
W20260808 15:59:40.699738 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:59:40 dist-test-slave-srgv krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:59:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:59:40.940659 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40701
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:41.267886 13401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:41.268205 13401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:41.268363 13401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:41.275266 13401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:41.275385 13401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:41.275480 13401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:41.275571 13401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:41.275655 13401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:41.284754 13401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:41.285925 13401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:41.287176 13401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:41.298434 13409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:41.298463 13407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:41.298691 13406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:41.299297 13401 server_base.cc:1034] running on GCE node
I20260808 15:59:41.300005 13401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:41.301307 13401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:41.302531 13401 hybrid_clock.cc:648] HybridClock initialized: now 1786204781302501 us; error 54 us; skew 500 ppm
I20260808 15:59:41.302848 13401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:41 dist-test-slave-srgv krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204781, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:41.305780 13401 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:59:41.307296 13401 webserver.cc:460] Webserver started at http://127.27.254.62:42567/ using document root <none> and password file <none>
I20260808 15:59:41.307888 13401 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:41.308000 13401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:41.308226 13401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:41.311060 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "11709073203d4d8b877b63cd236ebb32"
format_stamp: "Formatted at 2026-08-08 15:59:41 on dist-test-slave-srgv"
server_key: "62db27fd30e86a1289f06ae83e815507"
server_key_iv: "1e3874d0061f1c6301562fb5146c2b07"
server_key_version: "encryptionkey@0"
I20260808 15:59:41.311738 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "11709073203d4d8b877b63cd236ebb32"
format_stamp: "Formatted at 2026-08-08 15:59:41 on dist-test-slave-srgv"
server_key: "62db27fd30e86a1289f06ae83e815507"
server_key_iv: "1e3874d0061f1c6301562fb5146c2b07"
server_key_version: "encryptionkey@0"
I20260808 15:59:41.316411 13401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 15:59:41.319424 13416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:41.320586 13401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 15:59:41.320787 13401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "11709073203d4d8b877b63cd236ebb32"
format_stamp: "Formatted at 2026-08-08 15:59:41 on dist-test-slave-srgv"
server_key: "62db27fd30e86a1289f06ae83e815507"
server_key_iv: "1e3874d0061f1c6301562fb5146c2b07"
server_key_version: "encryptionkey@0"
I20260808 15:59:41.320976 13401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:41.332065 13401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:41.335559 13401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:41.335836 13401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:41.353380 13467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:40701 every 8 connection(s)
I20260808 15:59:41.353377 13401 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:40701
I20260808 15:59:41.354779 13401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:41.358565 13468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:41.363051 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13401
I20260808 15:59:41.363274 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:59:41.363889 28664 external_mini_cluster.cc:1428] Setting key 48f10dd71ac24038a3da40c214ab7f2d
I20260808 15:59:41.369596 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32: Bootstrap starting.
I20260808 15:59:41.373250 13468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:41.374262 13468 log.cc:824] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:41.377058 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32: No bootstrap required, opened a new log
Aug 08 15:59:41 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:41.386932 13468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11709073203d4d8b877b63cd236ebb32" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40701 } }
I20260808 15:59:41.387331 13468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:41.387439 13468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11709073203d4d8b877b63cd236ebb32, State: Initialized, Role: FOLLOWER
I20260808 15:59:41.388031 13468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11709073203d4d8b877b63cd236ebb32" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40701 } }
I20260808 15:59:41.388291 13468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:41.388324 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:41.365751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.27.254.62:40701)
0808 15:59:41.366590 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:41.366604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:41.366645 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:59:41.368841 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:41.368865 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:41.369301 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:41.369742 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:41.371139 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:41.373172 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:41.374699 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:41.375525 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:41.375717 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:41.379709 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:41.379720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:41.379737 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:41.379805 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:41.384021 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:41.384725 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:41.384732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:41.384740 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:41.384940 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:41.385377 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:41.385382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:41.385386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:41.386062 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:41.386340 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:41.386727 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 15:59:41.387165 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":355,"threads_started":1}
I20260808 15:59:41.388444 13468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:41.388666 13468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:41.391319 13468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11709073203d4d8b877b63cd236ebb32" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40701 } }
I20260808 15:59:41.391800 13468 leader_election.cc:302] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11709073203d4d8b877b63cd236ebb32; no voters:
I20260808 15:59:41.392361 13468 leader_election.cc:288] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:59:41.392452 13473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:41.393265 13473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [term 1 LEADER]: Becoming Leader. State: Replica: 11709073203d4d8b877b63cd236ebb32, State: Running, Role: LEADER
I20260808 15:59:41.393775 13473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11709073203d4d8b877b63cd236ebb32" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40701 } }
I20260808 15:59:41.394384 13468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:41.396150 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11709073203d4d8b877b63cd236ebb32. Latest consensus state: current_term: 1 leader_uuid: "11709073203d4d8b877b63cd236ebb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11709073203d4d8b877b63cd236ebb32" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40701 } } }
I20260808 15:59:41.396482 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:41.397990 13478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:41.397787 13474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11709073203d4d8b877b63cd236ebb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11709073203d4d8b877b63cd236ebb32" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40701 } } }
I20260808 15:59:41.398211 13474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:41.403745 13478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:41.412366 13478 catalog_manager.cc:1347] Generated new cluster ID: 8781b090c7f6461a8bd26060f2a8113a
I20260808 15:59:41.412492 13478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:41.427150 13478 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:59:41.428290 13478 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:41.441792 13478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32: Generated new TSK 0
I20260808 15:59:41.442482 13478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:59:41.505520 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40701
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:41.825893 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:41.826138 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:41.826226 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:41.833112 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:41.833204 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:41.833309 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:41.842554 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:41.843629 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:41.844866 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:41.857690 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:41.857645 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:41.857661 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:41.858580 13495 server_base.cc:1034] running on GCE node
I20260808 15:59:41.859151 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:41.860056 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:41.861290 13495 hybrid_clock.cc:648] HybridClock initialized: now 1786204781861252 us; error 50 us; skew 500 ppm
I20260808 15:59:41.861622 13495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:41 dist-test-slave-srgv krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204781, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:41.864650 13495 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:59:41.866003 13495 webserver.cc:460] Webserver started at http://127.27.254.1:35265/ using document root <none> and password file <none>
I20260808 15:59:41.866600 13495 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:41.866696 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:41.866950 13495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:41.869783 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "8b426c06fc7047f6b195509ed7947424"
format_stamp: "Formatted at 2026-08-08 15:59:41 on dist-test-slave-srgv"
server_key: "5441cd9de0520e04e2ab2c912f795317"
server_key_iv: "aa4a29d463fa3aeeccd8b61ff18e5382"
server_key_version: "encryptionkey@0"
I20260808 15:59:41.870466 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "8b426c06fc7047f6b195509ed7947424"
format_stamp: "Formatted at 2026-08-08 15:59:41 on dist-test-slave-srgv"
server_key: "5441cd9de0520e04e2ab2c912f795317"
server_key_iv: "aa4a29d463fa3aeeccd8b61ff18e5382"
server_key_version: "encryptionkey@0"
I20260808 15:59:41.875149 13495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:59:41.878399 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:41.879559 13495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 15:59:41.879684 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "8b426c06fc7047f6b195509ed7947424"
format_stamp: "Formatted at 2026-08-08 15:59:41 on dist-test-slave-srgv"
server_key: "5441cd9de0520e04e2ab2c912f795317"
server_key_iv: "aa4a29d463fa3aeeccd8b61ff18e5382"
server_key_version: "encryptionkey@0"
I20260808 15:59:41.879840 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:41.903391 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:41.907634 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:41.907872 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:41.908804 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:41.910082 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:41.910153 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:41.910295 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:41.910358 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:41.940990 13495 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:37295
I20260808 15:59:41.940991 13622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:37295 every 8 connection(s)
I20260808 15:59:41.942337 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:41.946507 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13495
I20260808 15:59:41.946741 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:59:41.947657 28664 external_mini_cluster.cc:1428] Setting key 7e6be7b7ca78242ec88106bb0553793d
Aug 08 15:59:41 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204781, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:41.967368 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:41.944855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:41557 (local address 127.27.254.62:40701)
0808 15:59:41.945378 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:41.945383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:41.950975 (+ 5592us) server_negotiation.cc:407] Connection header received
0808 15:59:41.953546 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:41.953554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:41.953640 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:41.953769 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:41.955161 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:41.956304 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:41.957834 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:41.958507 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:41.958652 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:41.961867 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:41.961872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:41.961875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:41.961905 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:41.964279 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:41.965180 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:41.965184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:41.965186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:41.965291 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:41.965769 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:41.965772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:41.965774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:41.966017 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:41.966126 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:41.966970 (+ 844us) server_negotiation.cc:299] Negotiation successful
0808 15:59:41.967130 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260808 15:59:41.968801 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:41.945450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40701 (local address 127.27.254.1:41557)
0808 15:59:41.950134 (+ 4684us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:41.950852 (+ 718us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:41.953243 (+ 2391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:41.953941 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:41.953949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:41.954462 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:41.954995 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:41.955010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:41.956461 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:41.956465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:41.957703 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:41.957714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:41.958704 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:41.958708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:41.958883 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:41.959527 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:41.959547 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:41.961683 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:41.964413 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:41.964424 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:41.964441 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:41.965068 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:41.965391 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:41.965398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:41.965423 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:41.965649 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:41.966157 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:41.966168 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:41.966570 (+ 402us) client_negotiation.cc:769] Sending connection context
0808 15:59:41.966884 (+ 314us) client_negotiation.cc:240] Negotiation successful
0808 15:59:41.967401 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4401,"thread_start_us":4161,"threads_started":1}
I20260808 15:59:41.970475 13623 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40701
I20260808 15:59:41.970795 13623 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:41.971477 13623 heartbeater.cc:511] Master 127.27.254.62:40701 requested a full tablet report, sending...
I20260808 15:59:41.973811 13433 ts_manager.cc:194] Registered new tserver with Master: 8b426c06fc7047f6b195509ed7947424 (127.27.254.1:37295)
I20260808 15:59:41.976011 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:41557
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:59:42.006927 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40701
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:42.317880 13629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:42.318166 13629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:42.318329 13629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:42.325480 13629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:42.325624 13629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:42.325786 13629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:42.335275 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:42.336496 13629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:42.337809 13629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:42.350417 13635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:42.350373 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:42.350448 13638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:42.351176 13629 server_base.cc:1034] running on GCE node
I20260808 15:59:42.351761 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:42.352564 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:42.353811 13629 hybrid_clock.cc:648] HybridClock initialized: now 1786204782353748 us; error 51 us; skew 500 ppm
I20260808 15:59:42.354153 13629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:42 dist-test-slave-srgv krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204782, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:42.357697 13629 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:59:42.362576 13629 webserver.cc:460] Webserver started at http://127.27.254.2:35129/ using document root <none> and password file <none>
I20260808 15:59:42.363238 13629 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:42.363368 13629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:42.363684 13629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:42.366703 13629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "8c49ec05412d4d5db2af765fc14add7a"
format_stamp: "Formatted at 2026-08-08 15:59:42 on dist-test-slave-srgv"
server_key: "34590116414b0fc7e31f3559f054eefc"
server_key_iv: "22cbe5b70710c5c5818d4af7760c6d3a"
server_key_version: "encryptionkey@0"
I20260808 15:59:42.367498 13629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "8c49ec05412d4d5db2af765fc14add7a"
format_stamp: "Formatted at 2026-08-08 15:59:42 on dist-test-slave-srgv"
server_key: "34590116414b0fc7e31f3559f054eefc"
server_key_iv: "22cbe5b70710c5c5818d4af7760c6d3a"
server_key_version: "encryptionkey@0"
I20260808 15:59:42.372618 13629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:59:42.375957 13645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:42.377221 13629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:59:42.377382 13629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "8c49ec05412d4d5db2af765fc14add7a"
format_stamp: "Formatted at 2026-08-08 15:59:42 on dist-test-slave-srgv"
server_key: "34590116414b0fc7e31f3559f054eefc"
server_key_iv: "22cbe5b70710c5c5818d4af7760c6d3a"
server_key_version: "encryptionkey@0"
I20260808 15:59:42.377612 13629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:42.404747 13629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:42.408528 13629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:42.408838 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:42.409782 13629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:42.411032 13629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:42.411158 13629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:42.411295 13629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:42.411376 13629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:42.444787 13629 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:38033
I20260808 15:59:42.444784 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:38033 every 8 connection(s)
I20260808 15:59:42.446138 13629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:42.447682 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13629
I20260808 15:59:42.447883 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:59:42.448381 28664 external_mini_cluster.cc:1428] Setting key 1e732b3c6b6125edc9351f73da7ec4d6
Aug 08 15:59:42 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204782, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:42.470086 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:42.448825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:36475 (local address 127.27.254.62:40701)
0808 15:59:42.449042 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:42.449047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:42.450462 (+ 1415us) server_negotiation.cc:407] Connection header received
0808 15:59:42.453297 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:42.453301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:42.453361 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:42.453516 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:42.455509 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.456195 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.458705 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.459129 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.459220 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:42.463243 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:42.463251 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:42.463259 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:42.463308 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:42.466873 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:42.467576 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:42.467584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:42.467590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:42.467754 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:42.468151 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:42.468158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:42.468164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:42.468528 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:42.468683 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:42.469328 (+ 645us) server_negotiation.cc:299] Negotiation successful
0808 15:59:42.469640 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:59:42.470816 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:42.449383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40701 (local address 127.27.254.2:36475)
0808 15:59:42.450270 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:42.450333 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:42.452979 (+ 2646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:42.453730 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:42.453744 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:42.454539 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:42.455326 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:42.455353 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.456337 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.456345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:42.458561 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:42.458579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.459349 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.459358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:42.459675 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:42.460411 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:42.460444 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:42.463021 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:42.467022 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:42.467029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:42.467041 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:42.467440 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:42.467859 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:42.467862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:42.467864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:42.468031 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:42.468645 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:42.468650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:42.468950 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 15:59:42.469152 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 15:59:42.469725 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":330,"threads_started":1}
I20260808 15:59:42.472294 13758 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40701
I20260808 15:59:42.472632 13758 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:42.473381 13758 heartbeater.cc:511] Master 127.27.254.62:40701 requested a full tablet report, sending...
I20260808 15:59:42.474915 13433 ts_manager.cc:194] Registered new tserver with Master: 8c49ec05412d4d5db2af765fc14add7a (127.27.254.2:38033)
I20260808 15:59:42.475666 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:36475
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:59:42.508828 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40701
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:42.833978 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:42.834247 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:42.834445 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:42.841631 13763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:42.841750 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:42.841931 13763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:42.851473 13763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:42.852741 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:42.854224 13763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:42.867082 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:42.866995 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:42.867053 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:42.868052 13763 server_base.cc:1034] running on GCE node
I20260808 15:59:42.868583 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:42.869540 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:42.870750 13763 hybrid_clock.cc:648] HybridClock initialized: now 1786204782870712 us; error 33 us; skew 500 ppm
I20260808 15:59:42.871058 13763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:42 dist-test-slave-srgv krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204782, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:42.874212 13763 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:59:42.875630 13763 webserver.cc:460] Webserver started at http://127.27.254.3:40041/ using document root <none> and password file <none>
I20260808 15:59:42.876250 13763 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:42.876348 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:42.876595 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:42.879557 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "b38c9b2f32e04806888d3d712cba3d8c"
format_stamp: "Formatted at 2026-08-08 15:59:42 on dist-test-slave-srgv"
server_key: "044629213b198af45a716d451f386a90"
server_key_iv: "20c0353758fc357f7e930048e036e3b0"
server_key_version: "encryptionkey@0"
I20260808 15:59:42.880273 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "b38c9b2f32e04806888d3d712cba3d8c"
format_stamp: "Formatted at 2026-08-08 15:59:42 on dist-test-slave-srgv"
server_key: "044629213b198af45a716d451f386a90"
server_key_iv: "20c0353758fc357f7e930048e036e3b0"
server_key_version: "encryptionkey@0"
I20260808 15:59:42.885725 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:59:42.888981 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:42.890331 13763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:42.890493 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "b38c9b2f32e04806888d3d712cba3d8c"
format_stamp: "Formatted at 2026-08-08 15:59:42 on dist-test-slave-srgv"
server_key: "044629213b198af45a716d451f386a90"
server_key_iv: "20c0353758fc357f7e930048e036e3b0"
server_key_version: "encryptionkey@0"
I20260808 15:59:42.890690 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:42.906240 13763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:42.909965 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:42.910250 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:42.911183 13763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:42.912483 13763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:42.912554 13763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:42.912730 13763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:42.912781 13763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:42.948331 13763 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:43517
I20260808 15:59:42.948375 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:43517 every 8 connection(s)
I20260808 15:59:42.949621 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:42.953369 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13763
I20260808 15:59:42.953593 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:59:42.954067 28664 external_mini_cluster.cc:1428] Setting key 2e6c030b1133a0de705b476f351240ba
Aug 08 15:59:42 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204782, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:42.968748 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:42.951974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:43911 (local address 127.27.254.62:40701)
0808 15:59:42.952186 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:42.952193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:42.953199 (+ 1006us) server_negotiation.cc:407] Connection header received
0808 15:59:42.955356 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:42.955361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:42.955446 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:42.955561 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:42.957258 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.958083 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.959569 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.959978 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.960111 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:42.963347 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:42.963359 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:42.963362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:42.963393 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:42.965743 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:42.966420 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:42.966442 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:42.966445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:42.966575 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:42.966998 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:42.967003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:42.967007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:42.967262 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:42.967410 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:42.968208 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 15:59:42.968406 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:59:42.969848 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:42.952382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:40701 (local address 127.27.254.3:43911)
0808 15:59:42.953045 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:42.953081 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:42.955039 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:42.955801 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:42.955809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:42.956429 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:42.957093 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:42.957111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.958237 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.958241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:42.959419 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:42.959430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.960212 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.960219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:42.960443 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:42.961010 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:42.961031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:42.963144 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:42.965874 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:42.965882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:42.965896 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:42.966284 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:42.966704 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:42.966709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:42.966712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:42.966888 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:42.967386 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:42.967394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:42.967803 (+ 409us) client_negotiation.cc:769] Sending connection context
0808 15:59:42.968101 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 15:59:42.968626 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260808 15:59:42.971158 13892 heartbeater.cc:347] Connected to a master server at 127.27.254.62:40701
I20260808 15:59:42.971494 13892 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:42.972172 13892 heartbeater.cc:511] Master 127.27.254.62:40701 requested a full tablet report, sending...
I20260808 15:59:42.973562 13433 ts_manager.cc:194] Registered new tserver with Master: b38c9b2f32e04806888d3d712cba3d8c (127.27.254.3:43517)
I20260808 15:59:42.974292 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:43911
I20260808 15:59:42.978796 13623 heartbeater.cc:503] Master 127.27.254.62:40701 was elected leader, sending a full tablet report...
I20260808 15:59:42.981339 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:59:42.997915 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:42.985927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.27.254.62:40701)
0808 15:59:42.986089 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:42.986094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:42.986501 (+ 407us) server_negotiation.cc:407] Connection header received
0808 15:59:42.986645 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:42.986648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:42.986701 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:42.986779 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:42.988073 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.988777 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.990102 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:42.990504 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:42.990597 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:42.992923 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:42.992927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:42.992930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:42.992958 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:42.995792 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:42.996357 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:42.996361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:42.996364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:42.996475 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:42.996748 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:42.996751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:42.996753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:42.997034 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:42.997220 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:42.997541 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 15:59:42.997701 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:59:43.001806 13433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:59:43.008517 13433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:59:43.031913 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.019688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37295 (local address 127.0.0.1:60828)
0808 15:59:43.022512 (+ 2824us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:43.022552 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:43.022757 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:43.023718 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:43.023727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:43.023765 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:43.024149 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.024165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.026280 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.026286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.028704 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.028718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.030945 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.030954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.031156 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.031216 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 15:59:43.031342 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 15:59:43.031518 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2638,"thread_start_us":3381,"threads_started":1}
I20260808 15:59:43.032820 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.019982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.27.254.1:37295)
0808 15:59:43.022092 (+ 2110us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.022099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.022680 (+ 581us) server_negotiation.cc:407] Connection header received
0808 15:59:43.022989 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.023001 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.023249 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.023451 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:43.024400 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.026072 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.028890 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.030414 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.030550 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.030795 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.032314 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.032492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1957,"thread_start_us":1792,"threads_started":1}
I20260808 15:59:43.034500 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.019872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:43517 (local address 127.0.0.1:43042)
0808 15:59:43.023334 (+ 3462us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:43.023362 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:43.023471 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:43.025384 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:43.025389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:43.025434 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:43.025840 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.025851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.028874 (+ 3023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.028879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.031925 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.031943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.033684 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.033691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.033862 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.033922 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 15:59:43.034004 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 15:59:43.034147 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3339,"thread_start_us":2353,"threads_started":1}
I20260808 15:59:43.034618 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.020529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.27.254.3:43517)
0808 15:59:43.024521 (+ 3992us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.024530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.024565 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:59:43.024664 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.024671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.024942 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.025142 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:43.026048 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.028717 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.032117 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.033471 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.033630 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.033851 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.034104 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.034269 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3814,"thread_start_us":3666,"threads_started":1}
I20260808 15:59:43.037078 13558 tablet_service.cc:1467] Processing CreateTablet for tablet f3485d400d7541f1b644a677bbf364c0 (DEFAULT_TABLE table=test-table [id=7bc91d09dcc14651a1e06782fe0858fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:43.038045 13827 tablet_service.cc:1467] Processing CreateTablet for tablet f3485d400d7541f1b644a677bbf364c0 (DEFAULT_TABLE table=test-table [id=7bc91d09dcc14651a1e06782fe0858fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:43.038651 13558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3485d400d7541f1b644a677bbf364c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:43.039451 13827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3485d400d7541f1b644a677bbf364c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:43.040335 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.024603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:38033 (local address 127.0.0.1:53648)
0808 15:59:43.033065 (+ 8462us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:43.033088 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:43.033206 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:43.033955 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:43.033960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:43.033986 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:43.034375 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.034387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.036881 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.036888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.038278 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.038289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.039592 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.039599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.039779 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.039802 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:59:43.039905 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:59:43.040026 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8331,"thread_start_us":8202,"threads_started":1}
I20260808 15:59:43.040385 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.023727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.27.254.2:38033)
0808 15:59:43.032760 (+ 9033us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.032782 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.033351 (+ 569us) server_negotiation.cc:407] Connection header received
0808 15:59:43.033473 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.033480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.033723 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.034585 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:43.034835 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.036750 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.038427 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.039389 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.039540 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.039723 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.039946 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.040095 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8876,"thread_start_us":8778,"threads_started":1}
I20260808 15:59:43.043454 13693 tablet_service.cc:1467] Processing CreateTablet for tablet f3485d400d7541f1b644a677bbf364c0 (DEFAULT_TABLE table=test-table [id=7bc91d09dcc14651a1e06782fe0858fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:43.044489 13693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3485d400d7541f1b644a677bbf364c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:43.053196 13910 tablet_bootstrap.cc:492] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Bootstrap starting.
I20260808 15:59:43.055209 13911 tablet_bootstrap.cc:492] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Bootstrap starting.
I20260808 15:59:43.056012 13912 tablet_bootstrap.cc:492] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Bootstrap starting.
I20260808 15:59:43.057354 13910 tablet_bootstrap.cc:654] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:43.058812 13910 log.cc:824] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:43.059489 13912 tablet_bootstrap.cc:654] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:43.059492 13911 tablet_bootstrap.cc:654] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:43.060724 13911 log.cc:824] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:43.060757 13912 log.cc:824] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:43.061625 13910 tablet_bootstrap.cc:492] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: No bootstrap required, opened a new log
I20260808 15:59:43.061883 13910 ts_tablet_manager.cc:1397] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260808 15:59:43.066607 13911 tablet_bootstrap.cc:492] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: No bootstrap required, opened a new log
I20260808 15:59:43.066879 13911 ts_tablet_manager.cc:1397] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260808 15:59:43.067248 13912 tablet_bootstrap.cc:492] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: No bootstrap required, opened a new log
I20260808 15:59:43.067512 13912 ts_tablet_manager.cc:1397] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260808 15:59:43.070577 13910 raft_consensus.cc:348] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.070988 13910 raft_consensus.cc:374] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:43.071090 13910 raft_consensus.cc:729] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b426c06fc7047f6b195509ed7947424, State: Initialized, Role: FOLLOWER
I20260808 15:59:43.071686 13910 consensus_queue.cc:260] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.073251 13910 ts_tablet_manager.cc:1428] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260808 15:59:43.074951 13911 raft_consensus.cc:348] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.075255 13911 raft_consensus.cc:374] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:43.075366 13911 raft_consensus.cc:729] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38c9b2f32e04806888d3d712cba3d8c, State: Initialized, Role: FOLLOWER
I20260808 15:59:43.075901 13911 consensus_queue.cc:260] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.076869 13892 heartbeater.cc:503] Master 127.27.254.62:40701 was elected leader, sending a full tablet report...
I20260808 15:59:43.077164 13912 raft_consensus.cc:348] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.077554 13911 ts_tablet_manager.cc:1428] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260808 15:59:43.077579 13912 raft_consensus.cc:374] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:43.077680 13912 raft_consensus.cc:729] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c49ec05412d4d5db2af765fc14add7a, State: Initialized, Role: FOLLOWER
I20260808 15:59:43.078186 13912 consensus_queue.cc:260] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.079277 13758 heartbeater.cc:503] Master 127.27.254.62:40701 was elected leader, sending a full tablet report...
I20260808 15:59:43.079694 13912 ts_tablet_manager.cc:1428] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 15:59:43.121369 13917 raft_consensus.cc:482] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:43.121675 13917 raft_consensus.cc:504] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.123229 13917 leader_election.cc:288] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b426c06fc7047f6b195509ed7947424 (127.27.254.1:37295), 8c49ec05412d4d5db2af765fc14add7a (127.27.254.2:38033)
I20260808 15:59:43.128722 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.123616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:37295 (local address 127.27.254.3:59707)
0808 15:59:43.123817 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:43.123837 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:43.123963 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:43.124307 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:43.124311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:43.124337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:43.124529 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.124538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.125598 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.125604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.126926 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.126940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.128011 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.128017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.128206 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.128229 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:59:43.128336 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 15:59:43.128459 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:59:43.128731 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.123615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:59707 (local address 127.27.254.1:37295)
0808 15:59:43.123811 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.123815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.123908 (+ 93us) server_negotiation.cc:407] Connection header received
0808 15:59:43.124105 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.124110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.124175 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.124277 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:43.124712 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.125451 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.127080 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.127859 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.128025 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.128096 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.128371 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.128491 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 15:59:43.129809 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3485d400d7541f1b644a677bbf364c0" candidate_uuid: "b38c9b2f32e04806888d3d712cba3d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b426c06fc7047f6b195509ed7947424" is_pre_election: true
I20260808 15:59:43.130136 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.124308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:55477 (local address 127.27.254.2:38033)
0808 15:59:43.124516 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.124524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.124912 (+ 388us) server_negotiation.cc:407] Connection header received
0808 15:59:43.125091 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.125095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.125156 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.125245 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:43.125889 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.126588 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.128192 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.129235 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.129399 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.129485 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.129769 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.129885 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 15:59:43.130172 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.124049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:38033 (local address 127.27.254.3:55477)
0808 15:59:43.124818 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:43.124830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:43.124957 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:43.125258 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:43.125263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:43.125283 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:43.125705 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.125716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.126745 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.126750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.128037 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:43.128048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.129437 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.129444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:43.129588 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.129620 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:59:43.129744 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 15:59:43.129856 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":489,"threads_started":1}
I20260808 15:59:43.130383 13578 raft_consensus.cc:2454] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38c9b2f32e04806888d3d712cba3d8c in term 0.
I20260808 15:59:43.131031 13781 leader_election.cc:302] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b426c06fc7047f6b195509ed7947424, b38c9b2f32e04806888d3d712cba3d8c; no voters:
I20260808 15:59:43.131122 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3485d400d7541f1b644a677bbf364c0" candidate_uuid: "b38c9b2f32e04806888d3d712cba3d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c49ec05412d4d5db2af765fc14add7a" is_pre_election: true
I20260808 15:59:43.131415 13917 raft_consensus.cc:2781] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:59:43.131564 13917 raft_consensus.cc:482] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:59:43.131669 13917 raft_consensus.cc:3037] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:43.131608 13713 raft_consensus.cc:2454] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b38c9b2f32e04806888d3d712cba3d8c in term 0.
I20260808 15:59:43.133924 13917 raft_consensus.cc:504] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.134421 13917 leader_election.cc:288] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [CANDIDATE]: Term 1 election: Requested vote from peers 8b426c06fc7047f6b195509ed7947424 (127.27.254.1:37295), 8c49ec05412d4d5db2af765fc14add7a (127.27.254.2:38033)
I20260808 15:59:43.134871 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3485d400d7541f1b644a677bbf364c0" candidate_uuid: "b38c9b2f32e04806888d3d712cba3d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b426c06fc7047f6b195509ed7947424"
I20260808 15:59:43.134948 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3485d400d7541f1b644a677bbf364c0" candidate_uuid: "b38c9b2f32e04806888d3d712cba3d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c49ec05412d4d5db2af765fc14add7a"
I20260808 15:59:43.135046 13578 raft_consensus.cc:3037] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:43.135171 13713 raft_consensus.cc:3037] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:43.137058 13578 raft_consensus.cc:2454] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38c9b2f32e04806888d3d712cba3d8c in term 1.
I20260808 15:59:43.137168 13713 raft_consensus.cc:2454] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38c9b2f32e04806888d3d712cba3d8c in term 1.
I20260808 15:59:43.137375 13781 leader_election.cc:302] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b426c06fc7047f6b195509ed7947424, b38c9b2f32e04806888d3d712cba3d8c; no voters:
I20260808 15:59:43.137650 13917 raft_consensus.cc:2781] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:43.138054 13917 raft_consensus.cc:686] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 1 LEADER]: Becoming Leader. State: Replica: b38c9b2f32e04806888d3d712cba3d8c, State: Running, Role: LEADER
I20260808 15:59:43.138468 13917 consensus_queue.cc:237] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } }
I20260808 15:59:43.142691 13432 catalog_manager.cc:5495] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to b38c9b2f32e04806888d3d712cba3d8c (127.27.254.3). New cstate: current_term: 1 leader_uuid: "b38c9b2f32e04806888d3d712cba3d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b38c9b2f32e04806888d3d712cba3d8c" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 43517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 } health_report { overall_health: UNKNOWN } } }
I20260808 15:59:43.186527 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.179174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.27.254.3:43517)
0808 15:59:43.179418 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.179425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.179449 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:59:43.179705 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.179713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.179826 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.179997 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:43.180337 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.181206 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.183875 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.184338 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.184482 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.185143 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:43.185889 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:59:43.186057 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.186156 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.186243 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 15:59:43.194856 13624 tablet.cc:2369] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:43.196954 13578 raft_consensus.cc:1261] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 1 FOLLOWER]: Refusing update from remote peer b38c9b2f32e04806888d3d712cba3d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:43.196954 13713 raft_consensus.cc:1261] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 1 FOLLOWER]: Refusing update from remote peer b38c9b2f32e04806888d3d712cba3d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:43.197964 13917 consensus_queue.cc:1035] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c49ec05412d4d5db2af765fc14add7a" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 38033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:59:43.198081 13759 tablet.cc:2369] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:43.198361 13920 consensus_queue.cc:1035] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b426c06fc7047f6b195509ed7947424" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 37295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:59:43.201103 13893 tablet.cc:2369] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:43.232370 13431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:43.232692 13431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:43.236136 13431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11709073203d4d8b877b63cd236ebb32: Sending DeleteTablet for 3 replicas of tablet f3485d400d7541f1b644a677bbf364c0
I20260808 15:59:43.237198 13827 tablet_service.cc:1514] Processing DeleteTablet for tablet f3485d400d7541f1b644a677bbf364c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:43 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:43042
I20260808 15:59:43.237352 13558 tablet_service.cc:1514] Processing DeleteTablet for tablet f3485d400d7541f1b644a677bbf364c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:43 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260808 15:59:43.237569 13693 tablet_service.cc:1514] Processing DeleteTablet for tablet f3485d400d7541f1b644a677bbf364c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:43 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:53648
I20260808 15:59:43.238643 13933 tablet_replica.cc:330] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: stopping tablet replica
I20260808 15:59:43.238693 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13495
I20260808 15:59:43.239069 13933 raft_consensus.cc:2229] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:59:43.239773 13933 raft_consensus.cc:2258] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:43.242123 13933 ts_tablet_manager.cc:1905] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:43.249977 13935 tablet_replica.cc:330] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: stopping tablet replica
I20260808 15:59:43.250475 13935 raft_consensus.cc:2229] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:43.251037 13935 raft_consensus.cc:2258] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:43.251884 13934 tablet_replica.cc:330] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: stopping tablet replica
I20260808 15:59:43.252336 13933 ts_tablet_manager.cc:1918] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:43.252341 13934 raft_consensus.cc:2229] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:43.252501 13933 log.cc:1196] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/f3485d400d7541f1b644a677bbf364c0
I20260808 15:59:43.252810 13934 raft_consensus.cc:2258] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:43.252967 13933 ts_tablet_manager.cc:1939] T f3485d400d7541f1b644a677bbf364c0 P b38c9b2f32e04806888d3d712cba3d8c: Deleting consensus metadata
I20260808 15:59:43.254680 13417 catalog_manager.cc:4841] TS b38c9b2f32e04806888d3d712cba3d8c (127.27.254.3:43517): tablet f3485d400d7541f1b644a677bbf364c0 (table test-table [id=7bc91d09dcc14651a1e06782fe0858fb]) successfully deleted
I20260808 15:59:43.255002 13934 ts_tablet_manager.cc:1905] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:43.255888 13935 ts_tablet_manager.cc:1905] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:43.262037 13934 ts_tablet_manager.cc:1918] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:43.262199 13934 log.cc:1196] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/f3485d400d7541f1b644a677bbf364c0
Aug 08 15:59:43 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:59:43.262741 13934 ts_tablet_manager.cc:1939] T f3485d400d7541f1b644a677bbf364c0 P 8b426c06fc7047f6b195509ed7947424: Deleting consensus metadata
I20260808 15:59:43.262976 13935 ts_tablet_manager.cc:1918] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:43.263180 13935 log.cc:1196] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/f3485d400d7541f1b644a677bbf364c0
I20260808 15:59:43.263800 13935 ts_tablet_manager.cc:1939] T f3485d400d7541f1b644a677bbf364c0 P 8c49ec05412d4d5db2af765fc14add7a: Deleting consensus metadata
I20260808 15:59:43.264649 13418 catalog_manager.cc:4841] TS 8b426c06fc7047f6b195509ed7947424 (127.27.254.1:37295): tablet f3485d400d7541f1b644a677bbf364c0 (table test-table [id=7bc91d09dcc14651a1e06782fe0858fb]) successfully deleted
I20260808 15:59:43.265551 13418 catalog_manager.cc:4841] TS 8c49ec05412d4d5db2af765fc14add7a (127.27.254.2:38033): tablet f3485d400d7541f1b644a677bbf364c0 (table test-table [id=7bc91d09dcc14651a1e06782fe0858fb]) successfully deleted
I20260808 15:59:43.269640 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.239752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.27.254.1:37295)
0808 15:59:43.239933 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.239940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.252075 (+ 12135us) server_negotiation.cc:407] Connection header received
0808 15:59:43.252255 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.252260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.252345 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.252458 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:43.254377 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.255448 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.257339 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.257999 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.258090 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.263520 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:43.263531 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:43.263546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:43.263582 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:43.266814 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:43.267518 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:43.267526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:43.267530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:43.267689 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:43.268122 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:43.268127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:43.268130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:43.268466 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:43.268758 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.269047 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.269247 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:59:43.270313 13618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:43.360714 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13495
I20260808 15:59:43.383816 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13629
Aug 08 15:59:43 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:59:43.402769 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.385055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.27.254.2:38033)
0808 15:59:43.385242 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.385250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.385271 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:59:43.385335 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.385342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.385469 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.385573 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:43.387560 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.388689 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.390738 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.391158 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.391256 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.396485 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:43.396498 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:43.396513 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:43.396549 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:43.400055 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:43.400709 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:43.400720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:43.400723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:43.400884 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:43.401209 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:43.401214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:43.401218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:43.401702 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:43.401986 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.402192 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.402420 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:59:43.403568 13753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:43.492692 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13629
I20260808 15:59:43.510852 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13763
Aug 08 15:59:43 dist-test-slave-srgv krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:59:43.528651 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:43.511703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.27.254.3:43517)
0808 15:59:43.511906 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:43.511912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:43.511934 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:59:43.512081 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:43.512086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:43.512180 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:43.512318 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:43.513783 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.515064 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.516336 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:43.517025 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:43.517164 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:43.521518 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:43.521532 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:43.521551 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:43.521598 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:43.525789 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:43.526399 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:43.526411 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:43.526415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:43.526608 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:43.526977 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:43.526982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:43.526985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:43.527454 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:43.527796 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:43.527972 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 15:59:43.528219 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:59:43.529397 13887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:43.650063 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13763
I20260808 15:59:43.667296 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13401
I20260808 15:59:43.668488 13463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:43.779577 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13401
2026-08-08T15:59:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4179 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T15:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:59:43Z Disabled control of system clock
I20260808 15:59:43.860680 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35113
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:35113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:59:44.187749 13944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:44.188031 13944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:44.188114 13944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:44.195405 13944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:44.195515 13944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:44.195552 13944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:44.195608 13944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:44.195652 13944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:44.204792 13944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:35113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:44.206110 13944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:44.207870 13944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:44.219506 13949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:44.219540 13952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:44.219592 13950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:44.220381 13944 server_base.cc:1034] running on GCE node
I20260808 15:59:44.221094 13944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:44.222581 13944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:44.223784 13944 hybrid_clock.cc:648] HybridClock initialized: now 1786204784223743 us; error 31 us; skew 500 ppm
I20260808 15:59:44.224092 13944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:44.226414 13944 webserver.cc:460] Webserver started at http://127.27.254.62:36995/ using document root <none> and password file <none>
I20260808 15:59:44.227047 13944 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:44.227129 13944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:44.227383 13944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:44.230228 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "d3b7763cac4040cf9565d1ca42bbf6de"
format_stamp: "Formatted at 2026-08-08 15:59:44 on dist-test-slave-srgv"
server_key: "ad8a7247a0aba32d6da61079e8440ad6"
server_key_iv: "e0a702922469664cdc25de63a28cecbf"
server_key_version: "encryptionkey@0"
I20260808 15:59:44.230970 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "d3b7763cac4040cf9565d1ca42bbf6de"
format_stamp: "Formatted at 2026-08-08 15:59:44 on dist-test-slave-srgv"
server_key: "ad8a7247a0aba32d6da61079e8440ad6"
server_key_iv: "e0a702922469664cdc25de63a28cecbf"
server_key_version: "encryptionkey@0"
I20260808 15:59:44.236004 13944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:59:44.239303 13959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:44.240674 13944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:59:44.240859 13944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "d3b7763cac4040cf9565d1ca42bbf6de"
format_stamp: "Formatted at 2026-08-08 15:59:44 on dist-test-slave-srgv"
server_key: "ad8a7247a0aba32d6da61079e8440ad6"
server_key_iv: "e0a702922469664cdc25de63a28cecbf"
server_key_version: "encryptionkey@0"
I20260808 15:59:44.241096 13944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:44.262547 13944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:44.263401 13944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:44.263702 13944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:44.282219 13944 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:35113
I20260808 15:59:44.282238 14011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:35113 every 8 connection(s)
I20260808 15:59:44.283555 13944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:44.287403 14012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:44.293243 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 13944
I20260808 15:59:44.293452 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:59:44.293926 28664 external_mini_cluster.cc:1428] Setting key 87a0586d8a818907478c3a53c26e20fc
I20260808 15:59:44.299250 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de: Bootstrap starting.
I20260808 15:59:44.303277 14012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:44.304291 14012 log.cc:824] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:44.307348 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de: No bootstrap required, opened a new log
I20260808 15:59:44.313689 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:44.295924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.27.254.62:35113)
0808 15:59:44.297023 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:44.297037 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:44.297088 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:59:44.299467 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:44.299496 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:44.300089 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:44.300511 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:44.301119 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:44.303110 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:44.304686 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:44.305634 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:44.305856 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:44.310628 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:44.310638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:44.310667 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:44.311403 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:44.311509 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:44.311544 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:44.311681 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:44.311949 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 15:59:44.312311 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":324,"threads_started":1}
I20260808 15:59:44.316006 14012 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35113 } }
I20260808 15:59:44.316402 14012 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:44.316516 14012 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b7763cac4040cf9565d1ca42bbf6de, State: Initialized, Role: FOLLOWER
I20260808 15:59:44.317070 14012 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35113 } }
I20260808 15:59:44.317293 14012 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:44.317389 14012 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:44.317603 14012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:44.319766 14012 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35113 } }
I20260808 15:59:44.320369 14012 leader_election.cc:302] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3b7763cac4040cf9565d1ca42bbf6de; no voters:
I20260808 15:59:44.321058 14012 leader_election.cc:288] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:59:44.321123 14017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:44.321979 14017 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [term 1 LEADER]: Becoming Leader. State: Replica: d3b7763cac4040cf9565d1ca42bbf6de, State: Running, Role: LEADER
I20260808 15:59:44.322525 14017 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35113 } }
I20260808 15:59:44.323376 14012 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:44.325614 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3b7763cac4040cf9565d1ca42bbf6de. Latest consensus state: current_term: 1 leader_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35113 } } }
I20260808 15:59:44.326086 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:44.325623 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7763cac4040cf9565d1ca42bbf6de" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 35113 } } }
I20260808 15:59:44.327291 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:44.328553 14023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:44.333724 14023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:44.342828 14023 catalog_manager.cc:1347] Generated new cluster ID: 9e69130f6da844cb8156b7efa326d321
I20260808 15:59:44.342932 14023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:44.371459 14023 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:59:44.372726 14023 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:44.383150 14023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de: Generated new TSK 0
I20260808 15:59:44.383971 14023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:59:44.394532 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35113
--builtin_ntp_servers=127.27.254.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:59:44.757635 14036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:44.757897 14036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:44.758051 14036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:44.765266 14036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:44.765362 14036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:44.765583 14036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:44.774909 14036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:44.776193 14036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:44.778018 14036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:44.792176 14042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:44.792330 14044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:44.793198 14041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:44.793676 14036 server_base.cc:1034] running on GCE node
I20260808 15:59:44.794445 14036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:44.795760 14036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:44.796989 14036 hybrid_clock.cc:648] HybridClock initialized: now 1786204784796889 us; error 93 us; skew 500 ppm
I20260808 15:59:44.797324 14036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:44.799674 14036 webserver.cc:460] Webserver started at http://127.27.254.1:41373/ using document root <none> and password file <none>
I20260808 15:59:44.800390 14036 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:44.800527 14036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:44.800858 14036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:44.803828 14036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "3053434d4dd64979905045879d348416"
format_stamp: "Formatted at 2026-08-08 15:59:44 on dist-test-slave-srgv"
server_key: "e988abce6e5add84c741060d56841786"
server_key_iv: "5921827a1e53dc944f352f2ec9618b5a"
server_key_version: "encryptionkey@0"
I20260808 15:59:44.804631 14036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "3053434d4dd64979905045879d348416"
format_stamp: "Formatted at 2026-08-08 15:59:44 on dist-test-slave-srgv"
server_key: "e988abce6e5add84c741060d56841786"
server_key_iv: "5921827a1e53dc944f352f2ec9618b5a"
server_key_version: "encryptionkey@0"
I20260808 15:59:44.809851 14036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:59:44.813371 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:44.815107 14036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:44.815299 14036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "3053434d4dd64979905045879d348416"
format_stamp: "Formatted at 2026-08-08 15:59:44 on dist-test-slave-srgv"
server_key: "e988abce6e5add84c741060d56841786"
server_key_iv: "5921827a1e53dc944f352f2ec9618b5a"
server_key_version: "encryptionkey@0"
I20260808 15:59:44.815510 14036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:44.831277 14036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:44.832028 14036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:44.832321 14036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:44.833220 14036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:44.834709 14036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:44.834829 14036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:44.834954 14036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:44.835039 14036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:44.866125 14036 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:41725
I20260808 15:59:44.866173 14164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:41725 every 8 connection(s)
I20260808 15:59:44.867395 14036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:44.874869 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14036
I20260808 15:59:44.875039 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:59:44.875550 28664 external_mini_cluster.cc:1428] Setting key c3a281e44470f7aeed6b2c277cae3dac
I20260808 15:59:44.878568 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35113
--builtin_ntp_servers=127.27.254.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:59:44.880337 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:44.869983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:54517 (local address 127.27.254.62:35113)
0808 15:59:44.870423 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:44.870428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:44.871594 (+ 1166us) server_negotiation.cc:407] Connection header received
0808 15:59:44.873263 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:44.873267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:44.873332 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:44.873484 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:44.874497 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:44.875316 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:44.877245 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:44.877847 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:44.877981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:44.879365 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:44.879374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:44.879381 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:44.879406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:44.879461 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:44.879479 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:44.879575 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:44.879780 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 15:59:44.879893 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260808 15:59:44.881076 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:44.870576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35113 (local address 127.27.254.1:54517)
0808 15:59:44.871441 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:44.871480 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:44.873032 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:44.873700 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:44.873707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:44.873792 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:44.874353 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:44.874369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:44.875462 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:44.875469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:44.877115 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:44.877133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:44.878051 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:44.878056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:44.878252 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:44.879052 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:44.879055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:44.879095 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:44.879176 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:44.879586 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:44.879594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:44.879599 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:59:44.879734 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 15:59:44.880022 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20260808 15:59:44.882656 14165 heartbeater.cc:347] Connected to a master server at 127.27.254.62:35113
I20260808 15:59:44.882974 14165 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:44.883852 14165 heartbeater.cc:511] Master 127.27.254.62:35113 requested a full tablet report, sending...
I20260808 15:59:44.886159 13976 ts_manager.cc:194] Registered new tserver with Master: 3053434d4dd64979905045879d348416 (127.27.254.1:41725)
I20260808 15:59:44.887789 13976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.1:54517
W20260808 15:59:45.204494 14169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:45.204813 14169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:45.204981 14169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:45.212154 14169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:45.212299 14169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:45.212478 14169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:45.221694 14169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:45.222934 14169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:45.224843 14169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:45.237893 14175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:45.237923 14174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:45.237845 14177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:45.238827 14169 server_base.cc:1034] running on GCE node
I20260808 15:59:45.239416 14169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:45.240375 14169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:45.241613 14169 hybrid_clock.cc:648] HybridClock initialized: now 1786204785241562 us; error 60 us; skew 500 ppm
I20260808 15:59:45.241940 14169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:45.244097 14169 webserver.cc:460] Webserver started at http://127.27.254.2:38715/ using document root <none> and password file <none>
I20260808 15:59:45.244747 14169 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:45.244854 14169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:45.245159 14169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:45.247991 14169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "64deb242ac4042a180dcafbec9ac3bdc"
format_stamp: "Formatted at 2026-08-08 15:59:45 on dist-test-slave-srgv"
server_key: "309bd8ee6b64cf66dc0b681331d7127c"
server_key_iv: "7cbcb70123ed28c473e1325b7ac42a37"
server_key_version: "encryptionkey@0"
I20260808 15:59:45.248764 14169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "64deb242ac4042a180dcafbec9ac3bdc"
format_stamp: "Formatted at 2026-08-08 15:59:45 on dist-test-slave-srgv"
server_key: "309bd8ee6b64cf66dc0b681331d7127c"
server_key_iv: "7cbcb70123ed28c473e1325b7ac42a37"
server_key_version: "encryptionkey@0"
I20260808 15:59:45.253729 14169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 15:59:45.257005 14183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:45.258306 14169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:45.258486 14169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "64deb242ac4042a180dcafbec9ac3bdc"
format_stamp: "Formatted at 2026-08-08 15:59:45 on dist-test-slave-srgv"
server_key: "309bd8ee6b64cf66dc0b681331d7127c"
server_key_iv: "7cbcb70123ed28c473e1325b7ac42a37"
server_key_version: "encryptionkey@0"
I20260808 15:59:45.258706 14169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:45.276149 14169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:45.276968 14169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:45.277278 14169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:45.278334 14169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:45.279760 14169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:45.279891 14169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:45.280026 14169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:45.280124 14169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:45.314730 14169 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:33653
I20260808 15:59:45.314781 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:33653 every 8 connection(s)
I20260808 15:59:45.316058 14169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:45.326016 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14169
I20260808 15:59:45.326191 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:59:45.326647 28664 external_mini_cluster.cc:1428] Setting key 1ab1f2c4414ee54cf62142391bfd3856
I20260808 15:59:45.328755 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.318579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:60389 (local address 127.27.254.62:35113)
0808 15:59:45.318758 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:45.318762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:45.320236 (+ 1474us) server_negotiation.cc:407] Connection header received
0808 15:59:45.322122 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:45.322128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:45.322223 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:45.322337 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:45.323401 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.324153 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.325854 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.326366 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.326498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.327942 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:45.327949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:45.327954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:45.327976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:45.328012 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:45.328023 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:45.328120 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:45.328392 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 15:59:45.328489 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:59:45.329679 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35113
--builtin_ntp_servers=127.27.254.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:59:45.330132 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.319150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35113 (local address 127.27.254.2:60389)
0808 15:59:45.320068 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:45.320102 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:45.321845 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:45.322575 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:45.322583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:45.322671 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:45.323227 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.323243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.324311 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.324316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.325694 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.325714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.326584 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.326591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.326873 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.327608 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:45.327611 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:45.327668 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:45.327761 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:45.328118 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:45.328130 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:45.328140 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:59:45.328351 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 15:59:45.328728 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":320,"threads_started":1}
I20260808 15:59:45.331513 14297 heartbeater.cc:347] Connected to a master server at 127.27.254.62:35113
I20260808 15:59:45.331820 14297 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:45.332541 14297 heartbeater.cc:511] Master 127.27.254.62:35113 requested a full tablet report, sending...
I20260808 15:59:45.333918 13976 ts_manager.cc:194] Registered new tserver with Master: 64deb242ac4042a180dcafbec9ac3bdc (127.27.254.2:33653)
I20260808 15:59:45.334671 13976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.2:60389
W20260808 15:59:45.665800 14300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:45.666078 14300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:45.666250 14300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:45.673323 14300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:45.673466 14300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:45.673641 14300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:45.682660 14300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:35113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:45.683957 14300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:45.685823 14300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:45.698648 14305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:45.698596 14308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:45.698690 14306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:45.699743 14300 server_base.cc:1034] running on GCE node
I20260808 15:59:45.700337 14300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:45.701216 14300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:45.702426 14300 hybrid_clock.cc:648] HybridClock initialized: now 1786204785702387 us; error 64 us; skew 500 ppm
I20260808 15:59:45.702705 14300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:45.704960 14300 webserver.cc:460] Webserver started at http://127.27.254.3:40067/ using document root <none> and password file <none>
I20260808 15:59:45.705694 14300 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:45.705786 14300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:45.706046 14300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:45.708760 14300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8"
format_stamp: "Formatted at 2026-08-08 15:59:45 on dist-test-slave-srgv"
server_key: "de4e7fbc7d177054dd4c66468778a5ad"
server_key_iv: "9a075b6204f5d5ed476d62ef2b1e29ec"
server_key_version: "encryptionkey@0"
I20260808 15:59:45.709544 14300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8"
format_stamp: "Formatted at 2026-08-08 15:59:45 on dist-test-slave-srgv"
server_key: "de4e7fbc7d177054dd4c66468778a5ad"
server_key_iv: "9a075b6204f5d5ed476d62ef2b1e29ec"
server_key_version: "encryptionkey@0"
I20260808 15:59:45.714530 14300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 15:59:45.717890 14314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:45.719297 14300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:59:45.719424 14300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8"
format_stamp: "Formatted at 2026-08-08 15:59:45 on dist-test-slave-srgv"
server_key: "de4e7fbc7d177054dd4c66468778a5ad"
server_key_iv: "9a075b6204f5d5ed476d62ef2b1e29ec"
server_key_version: "encryptionkey@0"
I20260808 15:59:45.719653 14300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:45.743317 14300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:45.744124 14300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:45.744467 14300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:45.745570 14300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:45.747016 14300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:45.747113 14300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:45.747244 14300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:45.747335 14300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:45.780543 14300 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:34117
I20260808 15:59:45.780655 14427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:34117 every 8 connection(s)
I20260808 15:59:45.781842 14300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:45.785058 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14300
I20260808 15:59:45.785271 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:59:45.785920 28664 external_mini_cluster.cc:1428] Setting key f4645596573d5a7ef7664c6cad528f87
I20260808 15:59:45.795013 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.784399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:53563 (local address 127.27.254.62:35113)
0808 15:59:45.784595 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:45.784599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:45.786305 (+ 1706us) server_negotiation.cc:407] Connection header received
0808 15:59:45.788545 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:45.788551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:45.788647 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:45.788773 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:45.789960 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.790703 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.792222 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.792659 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.792795 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.794178 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:45.794185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:45.794190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:45.794217 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:45.794248 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:45.794260 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:45.794358 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:45.794585 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 15:59:45.794692 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:59:45.795923 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.785037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:35113 (local address 127.27.254.3:53563)
0808 15:59:45.786139 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:45.786176 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:45.788262 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:45.788997 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:45.789005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:45.789136 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:45.789797 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.789814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.790855 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.790862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.792063 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.792075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.792892 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.792899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.793139 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.793862 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:45.793865 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:45.793903 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:45.793979 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:45.794357 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:45.794371 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:45.794381 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:59:45.794560 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 15:59:45.794936 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":404,"threads_started":1}
I20260808 15:59:45.797171 14428 heartbeater.cc:347] Connected to a master server at 127.27.254.62:35113
I20260808 15:59:45.797509 14428 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:45.798243 14428 heartbeater.cc:511] Master 127.27.254.62:35113 requested a full tablet report, sending...
I20260808 15:59:45.799734 13976 ts_manager.cc:194] Registered new tserver with Master: 67e1b4cf1d554fee8e39de4ce0b47fc8 (127.27.254.3:34117)
I20260808 15:59:45.800736 13976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.3:53563
I20260808 15:59:45.801038 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:59:45.810468 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.805000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.27.254.62:35113)
0808 15:59:45.805155 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:45.805159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:45.805706 (+ 547us) server_negotiation.cc:407] Connection header received
0808 15:59:45.805840 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:45.805844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:45.805925 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:45.806003 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:45.806430 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.807115 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.808376 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.808756 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.808841 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.809953 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:45.809958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:45.809961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:45.809976 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:45.810003 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:45.810013 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:45.810080 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:45.810204 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 15:59:45.810268 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:59:45.813941 13976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:59:45.820665 13976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:59:45.846220 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.832546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:36116)
0808 15:59:45.834096 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:45.834164 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:45.834372 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:45.836361 (+ 1989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:45.836368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:45.836408 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:45.836839 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.836853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.839454 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.839460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.842565 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.842589 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.844891 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.844901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.845197 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.845282 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 15:59:45.845449 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 15:59:45.845732 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":2144,"threads_started":1}
I20260808 15:59:45.847380 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.832751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.27.254.1:41725)
0808 15:59:45.835710 (+ 2959us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:45.835725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:45.835749 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:59:45.835849 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:45.835856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:45.836128 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:45.837064 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:45.837356 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.839292 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.842947 (+ 3655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.844614 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.846534 (+ 1920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.846872 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:45.846946 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 15:59:45.847098 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2808,"thread_start_us":2689,"threads_started":1}
I20260808 15:59:45.851047 14099 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4a4e42dd264be68d15255aa07217b1 (DEFAULT_TABLE table=test-table [id=80adf428b06740419216e57bfb65f79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:45.852522 14099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4a4e42dd264be68d15255aa07217b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:45.853183 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.832804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:34117 (local address 127.0.0.1:60628)
0808 15:59:45.835023 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:45.835045 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:45.835181 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:45.846522 (+ 11341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:45.846526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:45.846550 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:45.846886 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.846904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.849062 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.849067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.851016 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.851028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.852395 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.852402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.852602 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.852636 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:59:45.852733 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 15:59:45.852867 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2041,"thread_start_us":1048,"threads_started":1}
I20260808 15:59:45.853727 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.833487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.27.254.3:34117)
0808 15:59:45.845863 (+ 12376us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:45.845876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:45.845916 (+ 40us) server_negotiation.cc:407] Connection header received
0808 15:59:45.846026 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:45.846036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:45.846312 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:45.847107 (+ 795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:45.847387 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.848909 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.851213 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.852174 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.852355 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.852614 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:45.852827 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 15:59:45.852986 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11816,"thread_start_us":12089,"threads_started":1}
I20260808 15:59:45.855602 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.836194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:33653 (local address 127.0.0.1:39262)
0808 15:59:45.845851 (+ 9657us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:45.845871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:45.845998 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:45.848161 (+ 2163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:45.848166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:45.848191 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:45.848551 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.848561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.850694 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.850700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.852733 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:45.852747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.854850 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.854856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:45.855037 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.855068 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:59:45.855169 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:59:45.855300 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9540,"thread_start_us":9868,"threads_started":1}
I20260808 15:59:45.855748 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:45.836774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.27.254.2:33653)
0808 15:59:45.847585 (+ 10811us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:45.847600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:45.847630 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:59:45.847723 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:45.847733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:45.847937 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:45.848728 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:45.848922 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.850530 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.852961 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:45.854653 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:45.854788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:45.855013 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:45.855220 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 15:59:45.855376 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10639,"thread_start_us":10092,"threads_started":1}
I20260808 15:59:45.857470 14362 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4a4e42dd264be68d15255aa07217b1 (DEFAULT_TABLE table=test-table [id=80adf428b06740419216e57bfb65f79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:45.858960 14362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4a4e42dd264be68d15255aa07217b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:45.859169 14231 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4a4e42dd264be68d15255aa07217b1 (DEFAULT_TABLE table=test-table [id=80adf428b06740419216e57bfb65f79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:45.860517 14231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4a4e42dd264be68d15255aa07217b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:45.865100 14446 tablet_bootstrap.cc:492] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: Bootstrap starting.
I20260808 15:59:45.868927 14446 tablet_bootstrap.cc:654] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:45.870471 14446 log.cc:824] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:45.882582 14448 tablet_bootstrap.cc:492] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Bootstrap starting.
I20260808 15:59:45.883364 14446 tablet_bootstrap.cc:492] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: No bootstrap required, opened a new log
I20260808 15:59:45.883663 14446 ts_tablet_manager.cc:1397] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.006s
I20260808 15:59:45.886230 14449 tablet_bootstrap.cc:492] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Bootstrap starting.
I20260808 15:59:45.886322 14448 tablet_bootstrap.cc:654] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:45.887591 14448 log.cc:824] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:45.889676 14449 tablet_bootstrap.cc:654] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:45.890636 14449 log.cc:824] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:45.890731 14448 tablet_bootstrap.cc:492] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: No bootstrap required, opened a new log
I20260808 15:59:45.891031 14448 ts_tablet_manager.cc:1397] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:59:45.891522 14165 heartbeater.cc:503] Master 127.27.254.62:35113 was elected leader, sending a full tablet report...
I20260808 15:59:45.895663 14446 raft_consensus.cc:348] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:45.896124 14446 raft_consensus.cc:374] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:45.896229 14446 raft_consensus.cc:729] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3053434d4dd64979905045879d348416, State: Initialized, Role: FOLLOWER
I20260808 15:59:45.897002 14449 tablet_bootstrap.cc:492] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: No bootstrap required, opened a new log
I20260808 15:59:45.896840 14446 consensus_queue.cc:260] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:45.897341 14449 ts_tablet_manager.cc:1397] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 15:59:45.898593 14446 ts_tablet_manager.cc:1428] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260808 15:59:45.900972 14448 raft_consensus.cc:348] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:45.901458 14448 raft_consensus.cc:374] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:45.901584 14448 raft_consensus.cc:729] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64deb242ac4042a180dcafbec9ac3bdc, State: Initialized, Role: FOLLOWER
I20260808 15:59:45.902174 14448 consensus_queue.cc:260] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:45.903294 14297 heartbeater.cc:503] Master 127.27.254.62:35113 was elected leader, sending a full tablet report...
I20260808 15:59:45.903753 14448 ts_tablet_manager.cc:1428] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 15:59:45.907203 14449 raft_consensus.cc:348] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:45.907668 14449 raft_consensus.cc:374] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:45.907763 14449 raft_consensus.cc:729] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e1b4cf1d554fee8e39de4ce0b47fc8, State: Initialized, Role: FOLLOWER
I20260808 15:59:45.908281 14449 consensus_queue.cc:260] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:45.909188 14428 heartbeater.cc:503] Master 127.27.254.62:35113 was elected leader, sending a full tablet report...
I20260808 15:59:45.909852 14449 ts_tablet_manager.cc:1428] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260808 15:59:46.033614 14429 tablet.cc:2369] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:59:46.068094 14298 tablet.cc:2369] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:46.092124 14454 raft_consensus.cc:482] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:46.092450 14454 raft_consensus.cc:504] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:46.094218 14454 leader_election.cc:288] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3053434d4dd64979905045879d348416 (127.27.254.1:41725), 64deb242ac4042a180dcafbec9ac3bdc (127.27.254.2:33653)
I20260808 15:59:46.100795 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.094593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.27.254.3:33789)
0808 15:59:46.094792 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.094811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:46.094939 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:46.095540 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:46.095546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:46.095591 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:46.095800 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:46.095814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.097062 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.097066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:46.099024 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:46.099053 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.100306 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.100310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:46.100418 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.100441 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:59:46.100511 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 15:59:46.100597 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 15:59:46.100946 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.094678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:33789 (local address 127.27.254.1:41725)
0808 15:59:46.094859 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:46.094863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:46.094883 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:59:46.095104 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:46.095109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:46.095208 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:46.095318 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:46.096006 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.096934 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.099198 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.100167 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.100326 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.100392 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:46.100563 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 15:59:46.100684 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:59:46.101734 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.095186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:36965 (local address 127.27.254.2:33653)
0808 15:59:46.095436 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:46.095442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:46.095887 (+ 445us) server_negotiation.cc:407] Connection header received
0808 15:59:46.096061 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:46.096067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:46.096154 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:46.096258 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:46.096727 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.097566 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.099674 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.100782 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.100919 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.100999 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:46.101439 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 15:59:46.101560 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:59:46.101801 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.095031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:33653 (local address 127.27.254.3:36965)
0808 15:59:46.095784 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.095802 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:46.095930 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:46.096266 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:46.096269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:46.096288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:46.096562 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:46.096568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.097727 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.097732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:46.099503 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:46.099513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.101048 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.101055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:46.101244 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.101278 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:59:46.101362 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:59:46.101500 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":359,"threads_started":1}
I20260808 15:59:46.101984 14119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4a4e42dd264be68d15255aa07217b1" candidate_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3053434d4dd64979905045879d348416" is_pre_election: true
I20260808 15:59:46.102643 14119 raft_consensus.cc:2454] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e1b4cf1d554fee8e39de4ce0b47fc8 in term 0.
I20260808 15:59:46.102865 14251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4a4e42dd264be68d15255aa07217b1" candidate_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64deb242ac4042a180dcafbec9ac3bdc" is_pre_election: true
I20260808 15:59:46.103338 14316 leader_election.cc:302] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3053434d4dd64979905045879d348416, 67e1b4cf1d554fee8e39de4ce0b47fc8; no voters:
I20260808 15:59:46.103408 14251 raft_consensus.cc:2454] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e1b4cf1d554fee8e39de4ce0b47fc8 in term 0.
I20260808 15:59:46.103734 14454 raft_consensus.cc:2781] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:59:46.103917 14454 raft_consensus.cc:482] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:59:46.104030 14454 raft_consensus.cc:3037] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:46.106268 14454 raft_consensus.cc:504] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:46.106813 14454 leader_election.cc:288] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 3053434d4dd64979905045879d348416 (127.27.254.1:41725), 64deb242ac4042a180dcafbec9ac3bdc (127.27.254.2:33653)
I20260808 15:59:46.107251 14119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4a4e42dd264be68d15255aa07217b1" candidate_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3053434d4dd64979905045879d348416"
I20260808 15:59:46.107302 14251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4a4e42dd264be68d15255aa07217b1" candidate_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64deb242ac4042a180dcafbec9ac3bdc"
I20260808 15:59:46.107439 14119 raft_consensus.cc:3037] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:46.107542 14251 raft_consensus.cc:3037] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:46.109697 14119 raft_consensus.cc:2454] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e1b4cf1d554fee8e39de4ce0b47fc8 in term 1.
I20260808 15:59:46.109807 14251 raft_consensus.cc:2454] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e1b4cf1d554fee8e39de4ce0b47fc8 in term 1.
I20260808 15:59:46.110117 14316 leader_election.cc:302] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3053434d4dd64979905045879d348416, 67e1b4cf1d554fee8e39de4ce0b47fc8; no voters:
I20260808 15:59:46.110388 14454 raft_consensus.cc:2781] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:46.110805 14454 raft_consensus.cc:686] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 67e1b4cf1d554fee8e39de4ce0b47fc8, State: Running, Role: LEADER
I20260808 15:59:46.111264 14454 consensus_queue.cc:237] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } }
I20260808 15:59:46.115959 13975 catalog_manager.cc:5495] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e1b4cf1d554fee8e39de4ce0b47fc8 (127.27.254.3). New cstate: current_term: 1 leader_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e1b4cf1d554fee8e39de4ce0b47fc8" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 34117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 } health_report { overall_health: UNKNOWN } } }
W20260808 15:59:46.119777 14166 tablet.cc:2369] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:46.136919 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.131721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.27.254.3:34117)
0808 15:59:46.131900 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:46.131905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:46.132019 (+ 114us) server_negotiation.cc:407] Connection header received
0808 15:59:46.132211 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:46.132214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:46.132298 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:46.132424 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:46.132745 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.133564 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.135008 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.135413 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.135511 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.135952 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:46.136466 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:59:46.136573 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:46.136680 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 15:59:46.136740 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:59:46.144935 14251 raft_consensus.cc:1261] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 1 FOLLOWER]: Refusing update from remote peer 67e1b4cf1d554fee8e39de4ce0b47fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:46.144933 14119 raft_consensus.cc:1261] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416 [term 1 FOLLOWER]: Refusing update from remote peer 67e1b4cf1d554fee8e39de4ce0b47fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:46.145964 14456 consensus_queue.cc:1035] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64deb242ac4042a180dcafbec9ac3bdc" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:46.146323 14454 consensus_queue.cc:1035] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3053434d4dd64979905045879d348416" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:46.157384 14459 mvcc.cc:205] Tried to move back new op lower bound from 7316294804039151616 to 7316294803915329536. Current Snapshot: MvccSnapshot[applied={T|T < 7316294804039151616}]
I20260808 15:59:46.157791 14460 mvcc.cc:205] Tried to move back new op lower bound from 7316294804039151616 to 7316294803915329536. Current Snapshot: MvccSnapshot[applied={T|T < 7316294804039151616}]
I20260808 15:59:46.159890 14461 mvcc.cc:205] Tried to move back new op lower bound from 7316294804039151616 to 7316294803915329536. Current Snapshot: MvccSnapshot[applied={T|T < 7316294804039151616}]
I20260808 15:59:46.170820 13976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:46.171095 13976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:46.174544 13976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3b7763cac4040cf9565d1ca42bbf6de: Sending DeleteTablet for 3 replicas of tablet 5b4a4e42dd264be68d15255aa07217b1
I20260808 15:59:46.175716 14362 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4a4e42dd264be68d15255aa07217b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:46 UTC) from {username='slave'} at 127.0.0.1:60628
I20260808 15:59:46.175747 14099 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4a4e42dd264be68d15255aa07217b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:46 UTC) from {username='slave'} at 127.0.0.1:36116
I20260808 15:59:46.175931 14231 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4a4e42dd264be68d15255aa07217b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:46 UTC) from {username='slave'} at 127.0.0.1:39262
I20260808 15:59:46.177754 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14036
I20260808 15:59:46.208169 14469 tablet_replica.cc:330] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: stopping tablet replica
I20260808 15:59:46.208773 14469 raft_consensus.cc:2229] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:59:46.211668 14469 raft_consensus.cc:2258] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:46.214574 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.178959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.27.254.1:41725)
0808 15:59:46.179145 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:46.179151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:46.202785 (+ 23634us) server_negotiation.cc:407] Connection header received
0808 15:59:46.203325 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:46.203333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:46.203457 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:46.203539 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:46.204196 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.205555 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.209148 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.209926 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.210027 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.212452 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:46.212463 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:46.212481 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:46.213311 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:46.213374 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:46.213438 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:46.213526 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:46.214056 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 15:59:46.214184 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:59:46.216125 14160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:46.217713 14469 ts_tablet_manager.cc:1905] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:46.219319 14471 tablet_replica.cc:330] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: stopping tablet replica
I20260808 15:59:46.223096 14471 raft_consensus.cc:2229] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:46.223758 14471 raft_consensus.cc:2258] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:46.226564 14471 ts_tablet_manager.cc:1905] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:46.227999 14469 ts_tablet_manager.cc:1918] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:46.228212 14469 log.cc:1196] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/5b4a4e42dd264be68d15255aa07217b1
I20260808 15:59:46.228858 14469 ts_tablet_manager.cc:1939] T 5b4a4e42dd264be68d15255aa07217b1 P 67e1b4cf1d554fee8e39de4ce0b47fc8: Deleting consensus metadata
I20260808 15:59:46.230772 13963 catalog_manager.cc:4841] TS 67e1b4cf1d554fee8e39de4ce0b47fc8 (127.27.254.3:34117): tablet 5b4a4e42dd264be68d15255aa07217b1 (table test-table [id=80adf428b06740419216e57bfb65f79d]) successfully deleted
I20260808 15:59:46.234263 14471 ts_tablet_manager.cc:1918] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:46.234431 14471 log.cc:1196] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/5b4a4e42dd264be68d15255aa07217b1
I20260808 15:59:46.235030 14471 ts_tablet_manager.cc:1939] T 5b4a4e42dd264be68d15255aa07217b1 P 64deb242ac4042a180dcafbec9ac3bdc: Deleting consensus metadata
I20260808 15:59:46.236871 13961 catalog_manager.cc:4841] TS 64deb242ac4042a180dcafbec9ac3bdc (127.27.254.2:33653): tablet 5b4a4e42dd264be68d15255aa07217b1 (table test-table [id=80adf428b06740419216e57bfb65f79d]) successfully deleted
I20260808 15:59:46.311106 14470 tablet_replica.cc:330] T 5b4a4e42dd264be68d15255aa07217b1 P 3053434d4dd64979905045879d348416: stopping tablet replica
I20260808 15:59:46.311519 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14036
W20260808 15:59:46.329033 13961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:41725 (error 108)
I20260808 15:59:46.329978 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14169
I20260808 15:59:46.341683 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.341046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38338)
0808 15:59:46.341318 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.341486 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260808 15:59:46.342128 13961 catalog_manager.cc:4584] TS 3053434d4dd64979905045879d348416 (127.27.254.1:41725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b4a4e42dd264be68d15255aa07217b1: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
I20260808 15:59:46.342612 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.331937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.27.254.2:33653)
0808 15:59:46.332159 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:46.332167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:46.332191 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:59:46.332279 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:46.332287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:46.332416 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:46.332537 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:46.333357 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.335087 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.337687 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.338649 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.338816 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.340451 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:46.340462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:46.340478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:46.341280 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:46.341343 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:46.341368 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:46.341558 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:46.342149 (+ 591us) server_negotiation.cc:299] Negotiation successful
0808 15:59:46.342271 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:59:46.343333 14292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:46.396063 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.395502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38346)
0808 15:59:46.395769 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.395878 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260808 15:59:46.397241 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.396840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38350)
0808 15:59:46.397004 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.397091 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:59:46.445312 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14169
I20260808 15:59:46.457760 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.457154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38356)
0808 15:59:46.457432 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.457556 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 15:59:46.459043 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.458553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38362)
0808 15:59:46.458772 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.458872 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 15:59:46.463299 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14300
I20260808 15:59:46.471001 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.464203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.27.254.3:34117)
0808 15:59:46.464376 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:46.464380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:46.464397 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:59:46.464499 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:46.464503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:46.464571 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:46.464654 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:46.465162 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.466025 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.467945 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:46.468347 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:46.468434 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:46.469617 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:46.469625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:46.469641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:46.470204 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:46.470253 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:46.470272 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:46.470356 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:46.470546 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 15:59:46.470622 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:59:46.471684 14423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:46.539358 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.538875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38364)
0808 15:59:46.539093 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.539178 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 15:59:46.540377 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.539991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38378)
0808 15:59:46.540196 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.540280 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260808 15:59:46.591027 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14300
I20260808 15:59:46.611816 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 13944
I20260808 15:59:46.612958 14007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:46.720618 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:46.720054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:41725 (local address 127.0.0.1:38386)
0808 15:59:46.720282 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:46.720369 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 15:59:46.720574 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 13944
2026-08-08T15:59:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2936 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:59:46 dist-test-slave-srgv krb5kdc[14480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:59:46 dist-test-slave-srgv krb5kdc[14480](info): set up 2 sockets
Aug 08 15:59:46 dist-test-slave-srgv krb5kdc[14480](info): commencing operation
krb5kdc: starting...
W20260808 15:59:47.823313 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:59:47 dist-test-slave-srgv krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:59:48.077889 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:39721
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:39721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:48.397343 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:48.397724 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:48.397893 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:48.405082 14493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:48.405229 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:48.405336 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:48.405468 14493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:48.405551 14493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:48.415010 14493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:39721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:39721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:48.416230 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:48.417506 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:48.428849 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:48.428890 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:48.428918 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:48.430068 14493 server_base.cc:1034] running on GCE node
I20260808 15:59:48.430902 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:48.432283 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:48.433516 14493 hybrid_clock.cc:648] HybridClock initialized: now 1786204788433469 us; error 48 us; skew 500 ppm
I20260808 15:59:48.433876 14493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:48 dist-test-slave-srgv krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204788, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:48.436795 14493 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:59:48.438297 14493 webserver.cc:460] Webserver started at http://127.27.254.62:34033/ using document root <none> and password file <none>
I20260808 15:59:48.438971 14493 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:48.439117 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:48.439412 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:48.442495 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "d6db7c1cc4604e5c8030803092d4e9b3"
format_stamp: "Formatted at 2026-08-08 15:59:48 on dist-test-slave-srgv"
server_key: "f6f79586ee7b84f03252c619986fda5e"
server_key_iv: "29acd2ef46e1a9fe9006ee6730482160"
server_key_version: "encryptionkey@0"
I20260808 15:59:48.443254 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "d6db7c1cc4604e5c8030803092d4e9b3"
format_stamp: "Formatted at 2026-08-08 15:59:48 on dist-test-slave-srgv"
server_key: "f6f79586ee7b84f03252c619986fda5e"
server_key_iv: "29acd2ef46e1a9fe9006ee6730482160"
server_key_version: "encryptionkey@0"
I20260808 15:59:48.448314 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:59:48.451711 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:48.452983 14493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 15:59:48.453166 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "d6db7c1cc4604e5c8030803092d4e9b3"
format_stamp: "Formatted at 2026-08-08 15:59:48 on dist-test-slave-srgv"
server_key: "f6f79586ee7b84f03252c619986fda5e"
server_key_iv: "29acd2ef46e1a9fe9006ee6730482160"
server_key_version: "encryptionkey@0"
I20260808 15:59:48.453373 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:48.482520 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:48.486471 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:48.486754 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:48.504997 14493 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:39721
I20260808 15:59:48.505016 14560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:39721 every 8 connection(s)
I20260808 15:59:48.506400 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:48.510046 14561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:48.512315 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14493
I20260808 15:59:48.512517 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:59:48.513034 28664 external_mini_cluster.cc:1428] Setting key dcddbfacc451aeda1878ec33b245f074
I20260808 15:59:48.521370 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3: Bootstrap starting.
I20260808 15:59:48.524497 14561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:48.525530 14561 log.cc:824] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:59:48 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:48.528573 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3: No bootstrap required, opened a new log
I20260808 15:59:48.535207 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:48.515027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.27.254.62:39721)
0808 15:59:48.515769 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:48.515782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:48.515820 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:59:48.517474 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:48.517505 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:48.517871 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:48.518218 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:48.519771 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:48.521032 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:48.522459 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:48.523173 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:48.523396 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:48.527950 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:48.527957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:48.527970 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:48.528008 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:48.531122 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:48.531964 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:48.531970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:48.531976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:48.532140 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:48.532599 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:48.532602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:48.532604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:48.533118 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:48.533333 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:48.533799 (+ 466us) server_negotiation.cc:299] Negotiation successful
0808 15:59:48.534148 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":326,"threads_started":1}
I20260808 15:59:48.537755 14561 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39721 } }
I20260808 15:59:48.538228 14561 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:48.538338 14561 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6db7c1cc4604e5c8030803092d4e9b3, State: Initialized, Role: FOLLOWER
I20260808 15:59:48.538921 14561 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39721 } }
I20260808 15:59:48.539155 14561 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:48.539301 14561 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:48.539469 14561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:48.541972 14561 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39721 } }
I20260808 15:59:48.542474 14561 leader_election.cc:302] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6db7c1cc4604e5c8030803092d4e9b3; no voters:
I20260808 15:59:48.543099 14561 leader_election.cc:288] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:59:48.543221 14566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:48.544412 14566 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [term 1 LEADER]: Becoming Leader. State: Replica: d6db7c1cc4604e5c8030803092d4e9b3, State: Running, Role: LEADER
I20260808 15:59:48.544894 14566 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39721 } }
I20260808 15:59:48.545042 14561 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:48.547724 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6db7c1cc4604e5c8030803092d4e9b3. Latest consensus state: current_term: 1 leader_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39721 } } }
I20260808 15:59:48.548309 14567 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6db7c1cc4604e5c8030803092d4e9b3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 39721 } } }
I20260808 15:59:48.548646 14567 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:48.548942 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:48.552284 14575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:48.557144 14575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:48.566624 14575 catalog_manager.cc:1347] Generated new cluster ID: b62f6519f69c453b8bb9ae461085e5f5
I20260808 15:59:48.566741 14575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:48.584314 14575 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:59:48.586187 14575 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:48.609117 14575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3: Generated new TSK 0
I20260808 15:59:48.610301 14575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:59:48.668937 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39721
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:48.995906 14587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:48.996297 14587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:48.996475 14587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:49.003718 14587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:49.003813 14587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:49.003959 14587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:49.013594 14587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:49.014817 14587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:49.016108 14587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:49.028896 14592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:49.028937 14595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:49.028960 14593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:49.030069 14587 server_base.cc:1034] running on GCE node
I20260808 15:59:49.030726 14587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:49.031635 14587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:49.032861 14587 hybrid_clock.cc:648] HybridClock initialized: now 1786204789032816 us; error 39 us; skew 500 ppm
I20260808 15:59:49.033165 14587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:49 dist-test-slave-srgv krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:49.036540 14587 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:59:49.037907 14587 webserver.cc:460] Webserver started at http://127.27.254.1:38285/ using document root <none> and password file <none>
I20260808 15:59:49.038548 14587 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:49.038693 14587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:49.039004 14587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:49.041800 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "3a9f7b576ade48838233794ce554ccff"
format_stamp: "Formatted at 2026-08-08 15:59:49 on dist-test-slave-srgv"
server_key: "d7a50dc24d750ea8c9422f781a7af73a"
server_key_iv: "32a13f16a6b2453f6319ed6497502873"
server_key_version: "encryptionkey@0"
I20260808 15:59:49.042552 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "3a9f7b576ade48838233794ce554ccff"
format_stamp: "Formatted at 2026-08-08 15:59:49 on dist-test-slave-srgv"
server_key: "d7a50dc24d750ea8c9422f781a7af73a"
server_key_iv: "32a13f16a6b2453f6319ed6497502873"
server_key_version: "encryptionkey@0"
I20260808 15:59:49.047426 14587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 15:59:49.050845 14602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:49.052074 14587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:49.052243 14587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "3a9f7b576ade48838233794ce554ccff"
format_stamp: "Formatted at 2026-08-08 15:59:49 on dist-test-slave-srgv"
server_key: "d7a50dc24d750ea8c9422f781a7af73a"
server_key_iv: "32a13f16a6b2453f6319ed6497502873"
server_key_version: "encryptionkey@0"
I20260808 15:59:49.052423 14587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:49.078390 14587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:49.082089 14587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:49.082376 14587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:49.083316 14587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:49.084486 14587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:49.084551 14587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:49.084625 14587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:49.084666 14587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:49.116328 14587 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:32947
I20260808 15:59:49.116376 14714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:32947 every 8 connection(s)
I20260808 15:59:49.117681 14587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:49.124063 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14587
I20260808 15:59:49.124250 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:59:49.124786 28664 external_mini_cluster.cc:1428] Setting key fd8f27e8675f2482e36805523050dd10
Aug 08 15:59:49 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:49.138316 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:49.120129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:57779 (local address 127.27.254.62:39721)
0808 15:59:49.120582 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:49.120589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:49.121530 (+ 941us) server_negotiation.cc:407] Connection header received
0808 15:59:49.123245 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:49.123250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:49.123314 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:49.123409 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:49.125309 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.126175 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.127872 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.128274 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.128415 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:49.132365 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:49.132370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:49.132373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:49.132405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:49.134945 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:49.135803 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:49.135809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:49.135814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:49.135967 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:49.136358 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:49.136366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:49.136372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:49.136760 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:49.136925 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:49.137569 (+ 644us) server_negotiation.cc:299] Negotiation successful
0808 15:59:49.137830 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
I20260808 15:59:49.139119 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:49.120660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:39721 (local address 127.27.254.1:57779)
0808 15:59:49.121333 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:49.121376 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:49.123008 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:49.123598 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:49.123605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:49.124378 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:49.125139 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:49.125163 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.126363 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.126367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:49.127719 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:49.127736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.128497 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.128506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:49.128749 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:49.129444 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:49.129479 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:49.132161 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:49.135086 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:49.135098 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:49.135117 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:49.135670 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:49.136082 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:49.136085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:49.136087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:49.136250 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:49.136889 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:49.136894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:49.137178 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 15:59:49.137399 (+ 221us) client_negotiation.cc:240] Negotiation successful
0808 15:59:49.137836 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":262,"threads_started":1}
I20260808 15:59:49.140722 14715 heartbeater.cc:347] Connected to a master server at 127.27.254.62:39721
I20260808 15:59:49.141040 14715 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:49.141929 14715 heartbeater.cc:511] Master 127.27.254.62:39721 requested a full tablet report, sending...
I20260808 15:59:49.144098 14525 ts_manager.cc:194] Registered new tserver with Master: 3a9f7b576ade48838233794ce554ccff (127.27.254.1:32947)
I20260808 15:59:49.145750 14525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:57779
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:59:49.181506 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39721
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:49.500109 14721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:49.500407 14721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:49.500576 14721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:49.508020 14721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:49.508179 14721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:49.508361 14721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:49.517851 14721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:49.519035 14721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:49.520270 14721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:49.533124 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:49.533053 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:49.533147 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:49.534024 14721 server_base.cc:1034] running on GCE node
I20260808 15:59:49.534583 14721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:49.535485 14721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:49.536679 14721 hybrid_clock.cc:648] HybridClock initialized: now 1786204789536614 us; error 64 us; skew 500 ppm
I20260808 15:59:49.536975 14721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:49 dist-test-slave-srgv krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:49.543402 14721 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:59:49.544718 14721 webserver.cc:460] Webserver started at http://127.27.254.2:46197/ using document root <none> and password file <none>
I20260808 15:59:49.545526 14721 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:49.545634 14721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:49.545926 14721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:49.548787 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "8278abc0ab694b588f7ee8dc7eaeedf1"
format_stamp: "Formatted at 2026-08-08 15:59:49 on dist-test-slave-srgv"
server_key: "28ad6ccf7596d9a847de97a17836c052"
server_key_iv: "09d57979c1c5de49d66fa2f5f8dd1350"
server_key_version: "encryptionkey@0"
I20260808 15:59:49.549667 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "8278abc0ab694b588f7ee8dc7eaeedf1"
format_stamp: "Formatted at 2026-08-08 15:59:49 on dist-test-slave-srgv"
server_key: "28ad6ccf7596d9a847de97a17836c052"
server_key_iv: "09d57979c1c5de49d66fa2f5f8dd1350"
server_key_version: "encryptionkey@0"
I20260808 15:59:49.554606 14721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:59:49.558158 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:49.559350 14721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:49.559473 14721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "8278abc0ab694b588f7ee8dc7eaeedf1"
format_stamp: "Formatted at 2026-08-08 15:59:49 on dist-test-slave-srgv"
server_key: "28ad6ccf7596d9a847de97a17836c052"
server_key_iv: "09d57979c1c5de49d66fa2f5f8dd1350"
server_key_version: "encryptionkey@0"
I20260808 15:59:49.559621 14721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:49.576136 14721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:49.580114 14721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:49.580335 14721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:49.581159 14721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:49.582464 14721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:49.582547 14721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:49.582629 14721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:49.582680 14721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:49.615922 14721 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:32939
I20260808 15:59:49.615964 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:32939 every 8 connection(s)
I20260808 15:59:49.617172 14721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:49.625145 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14721
I20260808 15:59:49.625335 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:59:49.625902 28664 external_mini_cluster.cc:1428] Setting key 028746e55fbcf3826df4bd8b521cea78
Aug 08 15:59:49 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:49.637238 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:49.619746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:38301 (local address 127.27.254.62:39721)
0808 15:59:49.619919 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:49.619923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:49.621043 (+ 1120us) server_negotiation.cc:407] Connection header received
0808 15:59:49.622943 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:49.622949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:49.623048 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:49.623155 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:49.624708 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.625852 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.627724 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.628159 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.628302 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:49.631668 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:49.631673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:49.631676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:49.631707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:49.634328 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:49.635029 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:49.635034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:49.635040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:49.635174 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:49.635596 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:49.635602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:49.635605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:49.635860 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:49.636018 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:49.636662 (+ 644us) server_negotiation.cc:299] Negotiation successful
0808 15:59:49.636891 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:59:49.638242 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:49.620203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:39721 (local address 127.27.254.2:38301)
0808 15:59:49.620888 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:49.620931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:49.622669 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:49.623380 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:49.623388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:49.623871 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:49.624547 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:49.624566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.626055 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.626059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:49.627582 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:49.627594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:49.628392 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:49.628399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:49.628623 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:49.629286 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:49.629312 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:49.631446 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:49.634465 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:49.634474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:49.634488 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:49.634897 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:49.635291 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:49.635295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:49.635299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:49.635483 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:49.635996 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:49.636004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:49.636321 (+ 317us) client_negotiation.cc:769] Sending connection context
0808 15:59:49.636581 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 15:59:49.637089 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":271,"threads_started":1}
I20260808 15:59:49.639591 14852 heartbeater.cc:347] Connected to a master server at 127.27.254.62:39721
I20260808 15:59:49.639911 14852 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:49.640715 14852 heartbeater.cc:511] Master 127.27.254.62:39721 requested a full tablet report, sending...
I20260808 15:59:49.642256 14525 ts_manager.cc:194] Registered new tserver with Master: 8278abc0ab694b588f7ee8dc7eaeedf1 (127.27.254.2:32939)
I20260808 15:59:49.643023 14525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:38301
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:59:49.682133 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39721
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:50.010064 14857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:50.010332 14857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:50.010425 14857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:50.017467 14857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:50.017585 14857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:50.017696 14857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:50.026986 14857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:39721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:50.028126 14857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:50.029358 14857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:50.042425 14862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:50.043076 14863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:50.043109 14865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:50.043182 14857 server_base.cc:1034] running on GCE node
I20260808 15:59:50.043848 14857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:50.044720 14857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:50.045969 14857 hybrid_clock.cc:648] HybridClock initialized: now 1786204790045918 us; error 40 us; skew 500 ppm
I20260808 15:59:50.046265 14857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:50 dist-test-slave-srgv krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204790, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:50.049198 14857 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:59:50.050674 14857 webserver.cc:460] Webserver started at http://127.27.254.3:46623/ using document root <none> and password file <none>
I20260808 15:59:50.051390 14857 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:50.051537 14857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:50.051859 14857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:50.054728 14857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "136189f1308f471d82ddc420ef78a0a5"
format_stamp: "Formatted at 2026-08-08 15:59:50 on dist-test-slave-srgv"
server_key: "bc38a7331caeb4b5bba7ab9288318e27"
server_key_iv: "2072a7a6632e7e51c903119ca08940bf"
server_key_version: "encryptionkey@0"
I20260808 15:59:50.055508 14857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "136189f1308f471d82ddc420ef78a0a5"
format_stamp: "Formatted at 2026-08-08 15:59:50 on dist-test-slave-srgv"
server_key: "bc38a7331caeb4b5bba7ab9288318e27"
server_key_iv: "2072a7a6632e7e51c903119ca08940bf"
server_key_version: "encryptionkey@0"
I20260808 15:59:50.060748 14857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 15:59:50.064314 14872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:50.065675 14857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:50.065848 14857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "136189f1308f471d82ddc420ef78a0a5"
format_stamp: "Formatted at 2026-08-08 15:59:50 on dist-test-slave-srgv"
server_key: "bc38a7331caeb4b5bba7ab9288318e27"
server_key_iv: "2072a7a6632e7e51c903119ca08940bf"
server_key_version: "encryptionkey@0"
I20260808 15:59:50.066046 14857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:50.082412 14857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:50.086354 14857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:50.086642 14857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:50.087580 14857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:50.088873 14857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:50.088974 14857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:50.089120 14857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:50.089205 14857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:50.124068 14857 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:40627
I20260808 15:59:50.124115 14985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:40627 every 8 connection(s)
I20260808 15:59:50.125320 14857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:50.135320 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 14857
I20260808 15:59:50.135511 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:59:50.135936 28664 external_mini_cluster.cc:1428] Setting key 96128d1936849e9f918d81b8a21ba40d
Aug 08 15:59:50 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204790, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:50.144307 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.128005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:40061 (local address 127.27.254.62:39721)
0808 15:59:50.128168 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.128172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.129216 (+ 1044us) server_negotiation.cc:407] Connection header received
0808 15:59:50.131106 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.131112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.131194 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.131324 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:50.132709 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.133518 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.135045 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.135499 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.135624 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.139406 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:50.139410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:50.139413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:50.139444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:50.141884 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.142508 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.142511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.142513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.142631 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.142975 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.142978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.142980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.143221 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:50.143341 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.143911 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.144058 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:59:50.145231 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.128395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:39721 (local address 127.27.254.3:40061)
0808 15:59:50.129066 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:50.129102 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:50.130788 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:50.131558 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:50.131566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:50.132028 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:50.132547 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.132562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.133665 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.133670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.134896 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.134908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.135723 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.135730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.135981 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.136646 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:50.136675 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:50.139214 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:50.142007 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:50.142013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:50.142022 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:50.142412 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:50.142720 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:50.142723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:50.142726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:50.142888 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:50.143338 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:50.143343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:50.143618 (+ 275us) client_negotiation.cc:769] Sending connection context
0808 15:59:50.143825 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 15:59:50.144153 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260808 15:59:50.146481 14986 heartbeater.cc:347] Connected to a master server at 127.27.254.62:39721
I20260808 15:59:50.146785 14986 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:50.147442 14986 heartbeater.cc:511] Master 127.27.254.62:39721 requested a full tablet report, sending...
I20260808 15:59:50.148698 14525 ts_manager.cc:194] Registered new tserver with Master: 136189f1308f471d82ddc420ef78a0a5 (127.27.254.3:40627)
I20260808 15:59:50.149513 14715 heartbeater.cc:503] Master 127.27.254.62:39721 was elected leader, sending a full tablet report...
I20260808 15:59:50.149910 14525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:40061
I20260808 15:59:50.151103 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:59:50.167671 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.154971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.27.254.62:39721)
0808 15:59:50.155108 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.155111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.155572 (+ 461us) server_negotiation.cc:407] Connection header received
0808 15:59:50.155750 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.155755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.155823 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.155930 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:50.157301 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.158037 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.159327 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.159707 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.159838 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.162485 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:50.162491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:50.162496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:50.162536 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:50.164800 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.165549 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.165556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.165559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.165711 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.166162 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.166166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.166170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.166549 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:50.166816 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.167206 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.167385 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:59:50.171548 14524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:59:50.178201 14524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:59:50.202142 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.189461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.27.254.3:40627)
0808 15:59:50.190281 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.190290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.191039 (+ 749us) server_negotiation.cc:407] Connection header received
0808 15:59:50.191117 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.191123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.191323 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.191525 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:50.192778 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.194533 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.197836 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.199987 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.200143 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.200409 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.201441 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.201666 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":464,"threads_started":1}
I20260808 15:59:50.203119 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.189265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:40627 (local address 127.0.0.1:45368)
0808 15:59:50.190613 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:50.190663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:50.190855 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:50.191840 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:50.191848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:50.191889 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:50.192497 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.192514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.194776 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.194784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.197602 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.197621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.200556 (+ 2935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.200565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.200784 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.200860 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 15:59:50.200980 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 15:59:50.201175 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":1422,"threads_started":1}
I20260808 15:59:50.207185 14921 tablet_service.cc:1467] Processing CreateTablet for tablet 13ae3611208d486dbd30548808570505 (DEFAULT_TABLE table=test-table [id=ab9a45d2908c42bca7bf0a633ab1f56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:50.208622 14921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ae3611208d486dbd30548808570505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:50.210213 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.189680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.27.254.2:32939)
0808 15:59:50.191260 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.191304 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.202045 (+ 10741us) server_negotiation.cc:407] Connection header received
0808 15:59:50.202276 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.202286 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.202547 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.202745 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:50.203464 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.205068 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.207642 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.209089 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.209230 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.209396 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.209721 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.209894 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1432,"thread_start_us":1633,"threads_started":1}
I20260808 15:59:50.210235 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.190246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:32939 (local address 127.0.0.1:50004)
0808 15:59:50.201910 (+ 11664us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:50.201929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:50.202077 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:50.202879 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:50.202884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:50.202908 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:50.203264 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.203289 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.205210 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.205215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.207471 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.207485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.209306 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.209312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.209487 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.209525 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:59:50.209671 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 15:59:50.209802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11560,"mutex_wait_us":1,"thread_start_us":11066,"threads_started":1}
I20260808 15:59:50.210845 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.191714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.27.254.1:32947)
0808 15:59:50.202750 (+ 11036us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.202758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.202785 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:59:50.202968 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.202976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.203205 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.203421 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:50.204256 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.205398 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.208091 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.209142 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.209299 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.209530 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.210343 (+ 813us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.210500 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10825,"thread_start_us":10914,"threads_started":1}
I20260808 15:59:50.211622 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.191483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:32947 (local address 127.0.0.1:38372)
0808 15:59:50.202674 (+ 11191us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:50.202696 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:50.202812 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:50.203689 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:50.203693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:50.203717 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:50.204078 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.204090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.205590 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.205597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.207948 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.207959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.209701 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.209706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.209872 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.209920 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 15:59:50.210003 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:59:50.210124 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11005,"mutex_wait_us":333,"thread_start_us":10186,"threads_started":1}
I20260808 15:59:50.214025 14786 tablet_service.cc:1467] Processing CreateTablet for tablet 13ae3611208d486dbd30548808570505 (DEFAULT_TABLE table=test-table [id=ab9a45d2908c42bca7bf0a633ab1f56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:50.215122 14786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ae3611208d486dbd30548808570505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:50.215456 14650 tablet_service.cc:1467] Processing CreateTablet for tablet 13ae3611208d486dbd30548808570505 (DEFAULT_TABLE table=test-table [id=ab9a45d2908c42bca7bf0a633ab1f56b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:50.216741 14650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ae3611208d486dbd30548808570505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:50.222914 15004 tablet_bootstrap.cc:492] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Bootstrap starting.
I20260808 15:59:50.226430 15004 tablet_bootstrap.cc:654] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:50.227996 15004 log.cc:824] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:50.230094 15005 tablet_bootstrap.cc:492] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Bootstrap starting.
I20260808 15:59:50.234223 15006 tablet_bootstrap.cc:492] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Bootstrap starting.
I20260808 15:59:50.234304 15005 tablet_bootstrap.cc:654] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:50.235347 15005 log.cc:824] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:50.238133 15006 tablet_bootstrap.cc:654] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:50.238807 15004 tablet_bootstrap.cc:492] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: No bootstrap required, opened a new log
I20260808 15:59:50.238930 15005 tablet_bootstrap.cc:492] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: No bootstrap required, opened a new log
I20260808 15:59:50.239084 15004 ts_tablet_manager.cc:1397] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260808 15:59:50.239188 15005 ts_tablet_manager.cc:1397] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260808 15:59:50.239346 15006 log.cc:824] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:50.244305 15006 tablet_bootstrap.cc:492] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: No bootstrap required, opened a new log
I20260808 15:59:50.244565 15006 ts_tablet_manager.cc:1397] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260808 15:59:50.246852 15004 raft_consensus.cc:348] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.247191 15004 raft_consensus.cc:374] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:50.247311 15004 raft_consensus.cc:729] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136189f1308f471d82ddc420ef78a0a5, State: Initialized, Role: FOLLOWER
I20260808 15:59:50.247880 15004 consensus_queue.cc:260] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.248904 14986 heartbeater.cc:503] Master 127.27.254.62:39721 was elected leader, sending a full tablet report...
I20260808 15:59:50.249580 15004 ts_tablet_manager.cc:1428] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260808 15:59:50.249908 15005 raft_consensus.cc:348] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.250345 15005 raft_consensus.cc:374] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:50.250435 15005 raft_consensus.cc:729] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8278abc0ab694b588f7ee8dc7eaeedf1, State: Initialized, Role: FOLLOWER
I20260808 15:59:50.251003 15005 consensus_queue.cc:260] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.256356 15006 raft_consensus.cc:348] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.256894 15006 raft_consensus.cc:374] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:50.257017 15006 raft_consensus.cc:729] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9f7b576ade48838233794ce554ccff, State: Initialized, Role: FOLLOWER
I20260808 15:59:50.257786 15006 consensus_queue.cc:260] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.259090 14852 heartbeater.cc:503] Master 127.27.254.62:39721 was elected leader, sending a full tablet report...
I20260808 15:59:50.259845 15005 ts_tablet_manager.cc:1428] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Time spent starting tablet: real 0.021s user 0.014s sys 0.007s
I20260808 15:59:50.259970 15006 ts_tablet_manager.cc:1428] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260808 15:59:50.292637 15011 raft_consensus.cc:482] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:50.292964 15011 raft_consensus.cc:504] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.294621 15011 leader_election.cc:288] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136189f1308f471d82ddc420ef78a0a5 (127.27.254.3:40627), 3a9f7b576ade48838233794ce554ccff (127.27.254.1:32947)
I20260808 15:59:50.301169 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.295110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:60409 (local address 127.27.254.3:40627)
0808 15:59:50.295354 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.295362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.295759 (+ 397us) server_negotiation.cc:407] Connection header received
0808 15:59:50.296041 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.296049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.296142 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.296258 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:50.296847 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.297637 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.299518 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.300266 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.300453 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.300527 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.300785 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.300916 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:59:50.301251 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.294987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:40627 (local address 127.27.254.2:60409)
0808 15:59:50.295630 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:50.295645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:50.295859 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:50.296335 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:50.296340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:50.296365 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:50.296698 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.296705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.297775 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.297781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.299366 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.299377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.300475 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.300481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.300631 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.300654 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:59:50.300760 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:59:50.300894 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":351,"threads_started":1}
I20260808 15:59:50.301651 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.296081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:42677 (local address 127.27.254.1:32947)
0808 15:59:50.296323 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.296328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.296517 (+ 189us) server_negotiation.cc:407] Connection header received
0808 15:59:50.296734 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.296738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.296793 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.296889 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:50.297394 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.298090 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.299896 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.300681 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.300860 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.300936 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.301230 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.301395 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 15:59:50.301820 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.295830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:32947 (local address 127.27.254.2:42677)
0808 15:59:50.296417 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:50.296433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:50.296587 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:50.296903 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:50.296907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:50.296929 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:50.297251 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.297259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.298231 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.298236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.299737 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:50.299749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.300905 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.300911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:50.301075 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.301097 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:59:50.301194 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 15:59:50.301483 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":258,"threads_started":1}
I20260808 15:59:50.302381 14941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ae3611208d486dbd30548808570505" candidate_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136189f1308f471d82ddc420ef78a0a5" is_pre_election: true
I20260808 15:59:50.302728 14670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ae3611208d486dbd30548808570505" candidate_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9f7b576ade48838233794ce554ccff" is_pre_election: true
I20260808 15:59:50.302879 14941 raft_consensus.cc:2454] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8278abc0ab694b588f7ee8dc7eaeedf1 in term 0.
I20260808 15:59:50.303194 14670 raft_consensus.cc:2454] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8278abc0ab694b588f7ee8dc7eaeedf1 in term 0.
I20260808 15:59:50.303537 14740 leader_election.cc:302] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136189f1308f471d82ddc420ef78a0a5, 8278abc0ab694b588f7ee8dc7eaeedf1; no voters:
I20260808 15:59:50.303884 15011 raft_consensus.cc:2781] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:59:50.303997 15011 raft_consensus.cc:482] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:59:50.304131 15011 raft_consensus.cc:3037] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:50.306329 15011 raft_consensus.cc:504] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.306888 15011 leader_election.cc:288] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [CANDIDATE]: Term 1 election: Requested vote from peers 136189f1308f471d82ddc420ef78a0a5 (127.27.254.3:40627), 3a9f7b576ade48838233794ce554ccff (127.27.254.1:32947)
I20260808 15:59:50.307334 14941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ae3611208d486dbd30548808570505" candidate_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136189f1308f471d82ddc420ef78a0a5"
I20260808 15:59:50.307412 14670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ae3611208d486dbd30548808570505" candidate_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9f7b576ade48838233794ce554ccff"
I20260808 15:59:50.307529 14941 raft_consensus.cc:3037] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:50.307597 14670 raft_consensus.cc:3037] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:50.309841 14670 raft_consensus.cc:2454] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8278abc0ab694b588f7ee8dc7eaeedf1 in term 1.
I20260808 15:59:50.309844 14941 raft_consensus.cc:2454] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8278abc0ab694b588f7ee8dc7eaeedf1 in term 1.
I20260808 15:59:50.310214 14740 leader_election.cc:302] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9f7b576ade48838233794ce554ccff, 8278abc0ab694b588f7ee8dc7eaeedf1; no voters:
I20260808 15:59:50.310511 15011 raft_consensus.cc:2781] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:50.310936 15011 raft_consensus.cc:686] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 1 LEADER]: Becoming Leader. State: Replica: 8278abc0ab694b588f7ee8dc7eaeedf1, State: Running, Role: LEADER
I20260808 15:59:50.311401 15011 consensus_queue.cc:237] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } }
I20260808 15:59:50.315765 14525 catalog_manager.cc:5495] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8278abc0ab694b588f7ee8dc7eaeedf1 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8278abc0ab694b588f7ee8dc7eaeedf1" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 } health_report { overall_health: UNKNOWN } } }
I20260808 15:59:50.353960 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.348621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.27.254.2:32939)
0808 15:59:50.348901 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.348906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.348923 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:59:50.349103 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.349107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.349178 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.349337 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:50.349736 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.350517 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.351777 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.352173 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.352275 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.352790 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:50.353318 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:59:50.353502 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.353604 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.353662 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 15:59:50.362032 14941 raft_consensus.cc:1261] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 1 FOLLOWER]: Refusing update from remote peer 8278abc0ab694b588f7ee8dc7eaeedf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:50.362131 14670 raft_consensus.cc:1261] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 1 FOLLOWER]: Refusing update from remote peer 8278abc0ab694b588f7ee8dc7eaeedf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:50.362994 15011 consensus_queue.cc:1035] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "136189f1308f471d82ddc420ef78a0a5" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:50.363276 15015 consensus_queue.cc:1035] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9f7b576ade48838233794ce554ccff" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:59:50.369165 14853 tablet.cc:2369] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:59:50.370018 14716 tablet.cc:2369] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:50.389325 14525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:50.389703 14525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:50.393137 14525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6db7c1cc4604e5c8030803092d4e9b3: Sending DeleteTablet for 3 replicas of tablet 13ae3611208d486dbd30548808570505
I20260808 15:59:50.394281 14786 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ae3611208d486dbd30548808570505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:50 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:50004
I20260808 15:59:50.394474 14921 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ae3611208d486dbd30548808570505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:50 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:45368
I20260808 15:59:50.394975 14650 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ae3611208d486dbd30548808570505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:50 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:38372
I20260808 15:59:50.397293 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14587
I20260808 15:59:50.408512 15032 tablet_replica.cc:330] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: stopping tablet replica
I20260808 15:59:50.408946 15032 raft_consensus.cc:2229] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:50.409474 15032 raft_consensus.cc:2258] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:50.410492 15030 tablet_replica.cc:330] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: stopping tablet replica
I20260808 15:59:50.410874 15030 raft_consensus.cc:2229] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:59:50.411486 15030 raft_consensus.cc:2258] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:50.412180 15031 tablet_replica.cc:330] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: stopping tablet replica
I20260808 15:59:50.412541 15031 raft_consensus.cc:2229] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:50.412708 15032 ts_tablet_manager.cc:1905] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:50.412971 15031 raft_consensus.cc:2258] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:50.413770 15030 ts_tablet_manager.cc:1905] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:50.414929 15031 ts_tablet_manager.cc:1905] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:50.419490 15032 ts_tablet_manager.cc:1918] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:50.419656 15032 log.cc:1196] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/13ae3611208d486dbd30548808570505
I20260808 15:59:50.420205 15032 ts_tablet_manager.cc:1939] T 13ae3611208d486dbd30548808570505 P 3a9f7b576ade48838233794ce554ccff: Deleting consensus metadata
I20260808 15:59:50.421548 15031 ts_tablet_manager.cc:1918] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:50.421705 15031 log.cc:1196] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/13ae3611208d486dbd30548808570505
I20260808 15:59:50.422143 14511 catalog_manager.cc:4841] TS 3a9f7b576ade48838233794ce554ccff (127.27.254.1:32947): tablet 13ae3611208d486dbd30548808570505 (table test-table [id=ab9a45d2908c42bca7bf0a633ab1f56b]) successfully deleted
I20260808 15:59:50.422281 15031 ts_tablet_manager.cc:1939] T 13ae3611208d486dbd30548808570505 P 136189f1308f471d82ddc420ef78a0a5: Deleting consensus metadata
I20260808 15:59:50.423852 14511 catalog_manager.cc:4841] TS 136189f1308f471d82ddc420ef78a0a5 (127.27.254.3:40627): tablet 13ae3611208d486dbd30548808570505 (table test-table [id=ab9a45d2908c42bca7bf0a633ab1f56b]) successfully deleted
Aug 08 15:59:50 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:59:50.425731 15030 ts_tablet_manager.cc:1918] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:50.425897 15030 log.cc:1196] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/13ae3611208d486dbd30548808570505
I20260808 15:59:50.426354 15030 ts_tablet_manager.cc:1939] T 13ae3611208d486dbd30548808570505 P 8278abc0ab694b588f7ee8dc7eaeedf1: Deleting consensus metadata
I20260808 15:59:50.428033 14512 catalog_manager.cc:4841] TS 8278abc0ab694b588f7ee8dc7eaeedf1 (127.27.254.2:32939): tablet 13ae3611208d486dbd30548808570505 (table test-table [id=ab9a45d2908c42bca7bf0a633ab1f56b]) successfully deleted
I20260808 15:59:50.432703 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.398684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.27.254.1:32947)
0808 15:59:50.398887 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.398894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.405870 (+ 6976us) server_negotiation.cc:407] Connection header received
0808 15:59:50.406147 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.406153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.406240 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.406313 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:50.409887 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.410862 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.418527 (+ 7665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.419150 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.419278 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.426223 (+ 6945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:50.426236 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:50.426254 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:50.426299 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:50.429894 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.430555 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.430564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.430568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.430730 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.431132 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.431137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.431142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.431588 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:50.431875 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.432084 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.432309 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:59:50.433467 14710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:50.565186 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14587
I20260808 15:59:50.582425 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14721
Aug 08 15:59:50 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:59:50.598228 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.583341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.27.254.2:32939)
0808 15:59:50.583536 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.583543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.583565 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:59:50.583638 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.583643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.583727 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.583842 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:50.585191 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.586278 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.587585 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.588056 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.588196 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.592331 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:50.592343 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:50.592357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:50.592395 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:50.595345 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.596059 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.596072 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.596074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.596218 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.596621 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.596626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.596631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.596970 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:50.597303 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.597627 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.597874 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:59:50.598958 14847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:50.713842 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14721
I20260808 15:59:50.732869 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14857
Aug 08 15:59:50 dist-test-slave-srgv krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:59:50.748494 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:50.733616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.27.254.3:40627)
0808 15:59:50.733785 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:50.733789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:50.733807 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:59:50.733968 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:50.733972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:50.734034 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:50.734123 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:50.735446 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.736166 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.737466 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:50.737867 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:50.737961 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:50.743178 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:50.743192 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:50.743206 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:50.743236 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:50.746163 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.746793 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.746800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.746802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.746937 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:50.747256 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:50.747259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:50.747261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:50.747627 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:50.747835 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:50.748097 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 15:59:50.748255 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:59:50.749162 14981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:50.837996 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14857
I20260808 15:59:50.855724 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 14493
I20260808 15:59:50.856818 14556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:50.964983 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 14493
2026-08-08T15:59:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4244 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T15:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:59:51Z Disabled control of system clock
I20260808 15:59:51.040200 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46681
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:46681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:59:51.357930 15041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:51.358186 15041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:51.358378 15041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:51.365279 15041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:51.365399 15041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:51.365485 15041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:51.365581 15041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:51.365658 15041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:51.374306 15041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:46681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:51.375450 15041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:51.377182 15041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:51.388553 15047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:51.388589 15050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:51.388641 15048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:51.389691 15041 server_base.cc:1034] running on GCE node
I20260808 15:59:51.390383 15041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:51.391598 15041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:51.392823 15041 hybrid_clock.cc:648] HybridClock initialized: now 1786204791392790 us; error 45 us; skew 500 ppm
I20260808 15:59:51.393215 15041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:51.395454 15041 webserver.cc:460] Webserver started at http://127.27.254.62:38253/ using document root <none> and password file <none>
I20260808 15:59:51.396111 15041 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:51.396243 15041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:51.396564 15041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:51.399399 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "a6a5f1ec8a694aa099bfc8790898df13"
format_stamp: "Formatted at 2026-08-08 15:59:51 on dist-test-slave-srgv"
server_key: "9b1070ef1a9f70da7968cac2559fb0b7"
server_key_iv: "01b01bf719d040cb9138ee66e852c117"
server_key_version: "encryptionkey@0"
I20260808 15:59:51.400184 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "a6a5f1ec8a694aa099bfc8790898df13"
format_stamp: "Formatted at 2026-08-08 15:59:51 on dist-test-slave-srgv"
server_key: "9b1070ef1a9f70da7968cac2559fb0b7"
server_key_iv: "01b01bf719d040cb9138ee66e852c117"
server_key_version: "encryptionkey@0"
I20260808 15:59:51.405078 15041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260808 15:59:51.408668 15056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:51.409886 15041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 15:59:51.410071 15041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "a6a5f1ec8a694aa099bfc8790898df13"
format_stamp: "Formatted at 2026-08-08 15:59:51 on dist-test-slave-srgv"
server_key: "9b1070ef1a9f70da7968cac2559fb0b7"
server_key_iv: "01b01bf719d040cb9138ee66e852c117"
server_key_version: "encryptionkey@0"
I20260808 15:59:51.410310 15041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:51.435596 15041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:51.436360 15041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:51.436632 15041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:51.454398 15041 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:46681
I20260808 15:59:51.454418 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:46681 every 8 connection(s)
I20260808 15:59:51.455648 15041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:51.459259 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:51.461138 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15041
I20260808 15:59:51.461366 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:59:51.461825 28664 external_mini_cluster.cc:1428] Setting key b13a5ac530b55af05342e0e87fb59a9d
I20260808 15:59:51.470266 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13: Bootstrap starting.
I20260808 15:59:51.473372 15108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:51.474561 15108 log.cc:824] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:51.474934 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:51.463687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.27.254.62:46681)
0808 15:59:51.464576 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:51.464586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:51.464628 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:59:51.466453 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:51.466473 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:51.466902 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:51.467286 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:51.467853 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:51.469147 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:51.470663 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:51.471361 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:51.471562 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:51.472727 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:51.472734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:51.472746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:51.473230 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:51.473322 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:51.473341 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:51.473494 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:51.473714 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 15:59:51.474015 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":324,"threads_started":1}
I20260808 15:59:51.477303 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13: No bootstrap required, opened a new log
I20260808 15:59:51.484251 15108 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a5f1ec8a694aa099bfc8790898df13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46681 } }
I20260808 15:59:51.484577 15108 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:51.484694 15108 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a5f1ec8a694aa099bfc8790898df13, State: Initialized, Role: FOLLOWER
I20260808 15:59:51.485242 15108 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a5f1ec8a694aa099bfc8790898df13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46681 } }
I20260808 15:59:51.485562 15108 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:51.485716 15108 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:51.485881 15108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:51.487893 15108 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a5f1ec8a694aa099bfc8790898df13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46681 } }
I20260808 15:59:51.488349 15108 leader_election.cc:302] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a5f1ec8a694aa099bfc8790898df13; no voters:
I20260808 15:59:51.488873 15108 leader_election.cc:288] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:59:51.489033 15113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:51.489815 15113 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [term 1 LEADER]: Becoming Leader. State: Replica: a6a5f1ec8a694aa099bfc8790898df13, State: Running, Role: LEADER
I20260808 15:59:51.490337 15113 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a5f1ec8a694aa099bfc8790898df13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46681 } }
I20260808 15:59:51.490939 15108 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:51.493304 15114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a5f1ec8a694aa099bfc8790898df13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a5f1ec8a694aa099bfc8790898df13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46681 } } }
I20260808 15:59:51.493798 15114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:51.493724 15115 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a5f1ec8a694aa099bfc8790898df13. Latest consensus state: current_term: 1 leader_uuid: "a6a5f1ec8a694aa099bfc8790898df13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a5f1ec8a694aa099bfc8790898df13" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46681 } } }
I20260808 15:59:51.494081 15115 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:51.494779 15122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:51.499387 15122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:51.507205 15122 catalog_manager.cc:1347] Generated new cluster ID: 95328dfb5877432582ac2caf0fd4b360
I20260808 15:59:51.507301 15122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:51.529489 15122 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:59:51.530864 15122 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:51.540400 15122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13: Generated new TSK 0
I20260808 15:59:51.541177 15122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:59:51.555400 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46681
--builtin_ntp_servers=127.27.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:59:51.878679 15132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:51.878917 15132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:51.879061 15132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:51.885929 15132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:51.886030 15132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:51.886198 15132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:51.895788 15132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:51.896947 15132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:51.898875 15132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:51.911671 15137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:51.911705 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:51.911659 15140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:51.912782 15132 server_base.cc:1034] running on GCE node
I20260808 15:59:51.913365 15132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:51.914388 15132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:51.915612 15132 hybrid_clock.cc:648] HybridClock initialized: now 1786204791915558 us; error 49 us; skew 500 ppm
I20260808 15:59:51.915915 15132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:51.918179 15132 webserver.cc:460] Webserver started at http://127.27.254.1:42581/ using document root <none> and password file <none>
I20260808 15:59:51.918865 15132 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:51.918994 15132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:51.919309 15132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:51.922108 15132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "eb8a443cb553474db44704342dde82cc"
format_stamp: "Formatted at 2026-08-08 15:59:51 on dist-test-slave-srgv"
server_key: "dcda2511be2fa386e1d8202825386ffb"
server_key_iv: "5f56c38d9f0b8aba6ce8e580047d2848"
server_key_version: "encryptionkey@0"
I20260808 15:59:51.922883 15132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "eb8a443cb553474db44704342dde82cc"
format_stamp: "Formatted at 2026-08-08 15:59:51 on dist-test-slave-srgv"
server_key: "dcda2511be2fa386e1d8202825386ffb"
server_key_iv: "5f56c38d9f0b8aba6ce8e580047d2848"
server_key_version: "encryptionkey@0"
I20260808 15:59:51.927561 15132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:59:51.930843 15146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:51.932044 15132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 15:59:51.932224 15132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "eb8a443cb553474db44704342dde82cc"
format_stamp: "Formatted at 2026-08-08 15:59:51 on dist-test-slave-srgv"
server_key: "dcda2511be2fa386e1d8202825386ffb"
server_key_iv: "5f56c38d9f0b8aba6ce8e580047d2848"
server_key_version: "encryptionkey@0"
I20260808 15:59:51.932421 15132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:51.958353 15132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:51.959067 15132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:51.959323 15132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:51.960177 15132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:51.961547 15132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:51.961634 15132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:51.961714 15132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:51.961774 15132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:51.993953 15132 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:45907
I20260808 15:59:51.994097 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:45907 every 8 connection(s)
I20260808 15:59:51.995325 15132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:51.999701 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15132
I20260808 15:59:51.999874 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:59:52.000342 28664 external_mini_cluster.cc:1428] Setting key f6f00f3b940589accbf20a020f1245d1
I20260808 15:59:52.003091 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46681
--builtin_ntp_servers=127.27.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:59:52.007998 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:51.997811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:45617 (local address 127.27.254.62:46681)
0808 15:59:51.998262 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:51.998267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:51.999315 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 15:59:52.001552 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.001558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.001659 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.001783 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:52.002905 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.003692 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.005202 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.005696 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.005829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.007166 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:52.007179 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:52.007183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:52.007200 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:52.007239 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:52.007251 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:52.007346 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.007559 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.007661 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":249,"threads_started":1}
I20260808 15:59:52.008931 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:51.998281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46681 (local address 127.27.254.1:45617)
0808 15:59:51.999166 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:51.999201 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:52.001244 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:52.002034 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:52.002048 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:52.002195 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:52.002739 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.002755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.003840 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.003845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.005046 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.005058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.005922 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.005932 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.006182 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.006863 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:52.006866 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:52.006903 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:52.006978 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:52.007346 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:52.007360 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:52.007370 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:59:52.007549 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 15:59:52.007929 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260808 15:59:52.010423 15260 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46681
I20260808 15:59:52.010671 15260 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:52.011286 15260 heartbeater.cc:511] Master 127.27.254.62:46681 requested a full tablet report, sending...
I20260808 15:59:52.013284 15073 ts_manager.cc:194] Registered new tserver with Master: eb8a443cb553474db44704342dde82cc (127.27.254.1:45907)
I20260808 15:59:52.014824 15073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.1:45617
W20260808 15:59:52.324342 15264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:52.324577 15264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:52.324721 15264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:52.331766 15264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:52.331856 15264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:52.332046 15264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:52.340567 15264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:52.341804 15264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:52.343458 15264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:52.355497 15269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:52.355562 15270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:52.355710 15272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:52.356521 15264 server_base.cc:1034] running on GCE node
I20260808 15:59:52.357088 15264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:52.357998 15264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:52.359225 15264 hybrid_clock.cc:648] HybridClock initialized: now 1786204792359189 us; error 52 us; skew 500 ppm
I20260808 15:59:52.359531 15264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:52.361720 15264 webserver.cc:460] Webserver started at http://127.27.254.2:41565/ using document root <none> and password file <none>
I20260808 15:59:52.362370 15264 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:52.362499 15264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:52.362864 15264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:52.365584 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "7886c34323a94bb7a850b3a838041d78"
format_stamp: "Formatted at 2026-08-08 15:59:52 on dist-test-slave-srgv"
server_key: "78ad956e18a1e9c3d48b6b6ccb3a0c29"
server_key_iv: "3d211cdd4026b1c0d22638f04f07f574"
server_key_version: "encryptionkey@0"
I20260808 15:59:52.366343 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "7886c34323a94bb7a850b3a838041d78"
format_stamp: "Formatted at 2026-08-08 15:59:52 on dist-test-slave-srgv"
server_key: "78ad956e18a1e9c3d48b6b6ccb3a0c29"
server_key_iv: "3d211cdd4026b1c0d22638f04f07f574"
server_key_version: "encryptionkey@0"
I20260808 15:59:52.371330 15264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:59:52.374585 15278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:52.375816 15264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:52.375989 15264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "7886c34323a94bb7a850b3a838041d78"
format_stamp: "Formatted at 2026-08-08 15:59:52 on dist-test-slave-srgv"
server_key: "78ad956e18a1e9c3d48b6b6ccb3a0c29"
server_key_iv: "3d211cdd4026b1c0d22638f04f07f574"
server_key_version: "encryptionkey@0"
I20260808 15:59:52.376202 15264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:52.399151 15264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:52.399919 15264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:52.400218 15264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:52.401142 15264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:52.402549 15264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:52.402643 15264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:52.402789 15264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:52.402878 15264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:52.435783 15264 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:46067
I20260808 15:59:52.435897 15391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:46067 every 8 connection(s)
I20260808 15:59:52.437078 15264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:52.442060 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15264
I20260808 15:59:52.442255 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:59:52.442677 28664 external_mini_cluster.cc:1428] Setting key 5287bf44328bc3e9fea14146e1102603
I20260808 15:59:52.445310 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46681
--builtin_ntp_servers=127.27.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:59:52.449746 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.439571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:45625 (local address 127.27.254.62:46681)
0808 15:59:52.439773 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:52.439778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:52.441179 (+ 1401us) server_negotiation.cc:407] Connection header received
0808 15:59:52.443262 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.443267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.443356 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.443485 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:52.444518 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.445241 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.447019 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.447424 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.447563 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.448879 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:52.448884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:52.448887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:52.448902 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:52.448931 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:52.448941 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:52.449026 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.449270 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.449369 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:59:52.450703 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.440125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46681 (local address 127.27.254.2:45625)
0808 15:59:52.441023 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:52.441061 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:52.443010 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:52.443717 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:52.443725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:52.443815 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:52.444335 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.444351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.445451 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.445459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.446867 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.446880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.447661 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.447669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.447894 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.448561 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:52.448563 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:52.448631 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:52.448702 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:52.449042 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:52.449054 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:52.449067 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 15:59:52.449252 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 15:59:52.449689 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":291,"threads_started":1}
I20260808 15:59:52.452179 15392 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46681
I20260808 15:59:52.452509 15392 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:52.453203 15392 heartbeater.cc:511] Master 127.27.254.62:46681 requested a full tablet report, sending...
I20260808 15:59:52.464869 15073 ts_manager.cc:194] Registered new tserver with Master: 7886c34323a94bb7a850b3a838041d78 (127.27.254.2:46067)
I20260808 15:59:52.466010 15073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.2:45625
W20260808 15:59:52.770684 15395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:52.770983 15395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:52.771155 15395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:52.778139 15395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:52.778273 15395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:52.778478 15395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:52.787325 15395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:52.788585 15395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:52.790383 15395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:52.803457 15403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:52.803479 15400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:52.803381 15401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:52.806859 15395 server_base.cc:1034] running on GCE node
I20260808 15:59:52.807468 15395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:52.808372 15395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:52.809612 15395 hybrid_clock.cc:648] HybridClock initialized: now 1786204792809559 us; error 54 us; skew 500 ppm
I20260808 15:59:52.809940 15395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:52.812109 15395 webserver.cc:460] Webserver started at http://127.27.254.3:37153/ using document root <none> and password file <none>
I20260808 15:59:52.812772 15395 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:52.812902 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:52.813220 15395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:52.816164 15395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "00238a78253a4ca9b4695d5784219952"
format_stamp: "Formatted at 2026-08-08 15:59:52 on dist-test-slave-srgv"
server_key: "e57636a6460e82898860ffee8d7f689f"
server_key_iv: "10ffa4bc2b2462521270b1407dee94dc"
server_key_version: "encryptionkey@0"
I20260808 15:59:52.816906 15395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "00238a78253a4ca9b4695d5784219952"
format_stamp: "Formatted at 2026-08-08 15:59:52 on dist-test-slave-srgv"
server_key: "e57636a6460e82898860ffee8d7f689f"
server_key_iv: "10ffa4bc2b2462521270b1407dee94dc"
server_key_version: "encryptionkey@0"
I20260808 15:59:52.821807 15395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:59:52.825191 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:52.826452 15395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:52.826622 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "00238a78253a4ca9b4695d5784219952"
format_stamp: "Formatted at 2026-08-08 15:59:52 on dist-test-slave-srgv"
server_key: "e57636a6460e82898860ffee8d7f689f"
server_key_iv: "10ffa4bc2b2462521270b1407dee94dc"
server_key_version: "encryptionkey@0"
I20260808 15:59:52.826848 15395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:52.849049 15395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:52.849900 15395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:52.850211 15395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:52.851133 15395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:52.852523 15395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:52.852635 15395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:52.852767 15395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:52.852859 15395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:52.883402 15395 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:33443
I20260808 15:59:52.883496 15522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:33443 every 8 connection(s)
I20260808 15:59:52.884694 15395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:52.890388 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15395
I20260808 15:59:52.890568 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:59:52.891156 28664 external_mini_cluster.cc:1428] Setting key cf5c1c8c6c24a8a3a24ad5c4a75542b5
I20260808 15:59:52.897820 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.887261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:45319 (local address 127.27.254.62:46681)
0808 15:59:52.887441 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:52.887446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:52.888777 (+ 1331us) server_negotiation.cc:407] Connection header received
0808 15:59:52.891012 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.891035 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.891123 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.891215 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:52.892495 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.893578 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.895340 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.895814 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.895912 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.897105 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:52.897110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:52.897113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:52.897129 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:52.897161 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:52.897172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:52.897240 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.897477 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.897573 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:59:52.898705 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.887777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:46681 (local address 127.27.254.3:45319)
0808 15:59:52.888628 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:52.888670 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:52.890723 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:52.891476 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:52.891487 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:52.891607 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:52.892317 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.892343 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.893727 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.893731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.895221 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.895233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.895995 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.895999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.896195 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.896825 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:52.896827 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:52.896862 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:52.896939 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:52.897253 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:52.897261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:52.897266 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:59:52.897395 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 15:59:52.897735 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":268,"threads_started":1}
I20260808 15:59:52.900048 15523 heartbeater.cc:347] Connected to a master server at 127.27.254.62:46681
I20260808 15:59:52.900336 15523 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:52.901016 15523 heartbeater.cc:511] Master 127.27.254.62:46681 requested a full tablet report, sending...
I20260808 15:59:52.902371 15073 ts_manager.cc:194] Registered new tserver with Master: 00238a78253a4ca9b4695d5784219952 (127.27.254.3:33443)
I20260808 15:59:52.903153 15073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.3:45319
I20260808 15:59:52.906333 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:59:52.915975 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.910327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.27.254.62:46681)
0808 15:59:52.910473 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:52.910477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:52.910977 (+ 500us) server_negotiation.cc:407] Connection header received
0808 15:59:52.911105 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.911108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.911169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.911245 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:52.911671 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.912330 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.913789 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.914223 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.914379 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.915418 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:52.915422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:52.915425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:52.915441 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:52.915468 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:52.915478 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:52.915580 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.915732 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.915773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:59:52.919695 15073 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:59:52.926429 15073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:59:52.948850 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.938197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:33443 (local address 127.0.0.1:36436)
0808 15:59:52.939066 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:52.939120 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:52.939300 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:52.941154 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:52.941165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:52.941204 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:52.941687 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.941700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.943661 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.943666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.946023 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.946036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.947902 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.947909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.948114 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.948170 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 15:59:52.948321 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 15:59:52.948507 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":411,"threads_started":1}
I20260808 15:59:52.948925 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.938521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.27.254.3:33443)
0808 15:59:52.940627 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:52.940633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:52.940659 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:59:52.940752 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.940757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.940928 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.941138 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:52.941917 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.943500 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.946240 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.947693 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.947832 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.948102 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.948383 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.948557 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1942,"thread_start_us":1798,"threads_started":1}
I20260808 15:59:52.949286 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.938304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:46067 (local address 127.0.0.1:50020)
0808 15:59:52.940091 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:52.940108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:52.940213 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:52.942371 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:52.942378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:52.942414 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:52.942932 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.942944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.944846 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.944852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.946646 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.946664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.948322 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.948329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.948499 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.948528 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:59:52.948606 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 15:59:52.948716 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"thread_start_us":4411,"threads_started":1}
I20260808 15:59:52.949548 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.938854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.27.254.2:46067)
0808 15:59:52.941765 (+ 2911us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:52.941771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:52.941792 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:59:52.941900 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.941907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.942148 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.942352 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:52.943130 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.944724 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.946901 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.947753 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.947906 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.948170 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.948942 (+ 772us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.949126 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2783,"thread_start_us":2683,"threads_started":1}
I20260808 15:59:52.951985 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.940156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.27.254.1:45907)
0808 15:59:52.940742 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:52.940753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:52.943226 (+ 2473us) server_negotiation.cc:407] Connection header received
0808 15:59:52.943316 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:52.943323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:52.943516 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:52.943699 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:52.944536 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.946034 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.948517 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.950251 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.950382 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.950624 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:52.951476 (+ 852us) server_negotiation.cc:299] Negotiation successful
0808 15:59:52.951657 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":294,"threads_started":1}
I20260808 15:59:52.952668 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:52.939528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59868)
0808 15:59:52.942987 (+ 3459us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:52.943006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:52.943117 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:52.943849 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:52.943854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:52.943882 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:52.944328 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.944340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.946213 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.946222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.948262 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:52.948304 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:52.950753 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:52.950760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:52.950921 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:52.950952 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:59:52.951044 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:59:52.952341 (+ 1297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3322,"mutex_wait_us":1,"thread_start_us":443,"threads_started":1}
I20260808 15:59:52.953310 15326 tablet_service.cc:1467] Processing CreateTablet for tablet d79691e5ed51493cadd88d766da53ba3 (DEFAULT_TABLE table=test-table [id=2e71b144f95c451f829c34e81a1d70b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:52.953312 15457 tablet_service.cc:1467] Processing CreateTablet for tablet d79691e5ed51493cadd88d766da53ba3 (DEFAULT_TABLE table=test-table [id=2e71b144f95c451f829c34e81a1d70b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:52.954560 15326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79691e5ed51493cadd88d766da53ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:52.954798 15457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79691e5ed51493cadd88d766da53ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:52.956065 15195 tablet_service.cc:1467] Processing CreateTablet for tablet d79691e5ed51493cadd88d766da53ba3 (DEFAULT_TABLE table=test-table [id=2e71b144f95c451f829c34e81a1d70b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:52.957312 15195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79691e5ed51493cadd88d766da53ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:52.967461 15542 tablet_bootstrap.cc:492] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Bootstrap starting.
I20260808 15:59:52.970166 15543 tablet_bootstrap.cc:492] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Bootstrap starting.
I20260808 15:59:52.970871 15542 tablet_bootstrap.cc:654] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:52.971534 15544 tablet_bootstrap.cc:492] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: Bootstrap starting.
I20260808 15:59:52.972628 15542 log.cc:824] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:52.973663 15543 tablet_bootstrap.cc:654] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:52.974547 15544 tablet_bootstrap.cc:654] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:52.975042 15543 log.cc:824] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:52.975502 15544 log.cc:824] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:52.978319 15544 tablet_bootstrap.cc:492] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: No bootstrap required, opened a new log
I20260808 15:59:52.978579 15544 ts_tablet_manager.cc:1397] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 15:59:52.978679 15542 tablet_bootstrap.cc:492] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: No bootstrap required, opened a new log
I20260808 15:59:52.978848 15543 tablet_bootstrap.cc:492] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: No bootstrap required, opened a new log
I20260808 15:59:52.978947 15542 ts_tablet_manager.cc:1397] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260808 15:59:52.979080 15543 ts_tablet_manager.cc:1397] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:59:52.986310 15544 raft_consensus.cc:348] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:52.986678 15544 raft_consensus.cc:374] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:52.986814 15544 raft_consensus.cc:729] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8a443cb553474db44704342dde82cc, State: Initialized, Role: FOLLOWER
I20260808 15:59:52.987421 15544 consensus_queue.cc:260] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:52.988492 15260 heartbeater.cc:503] Master 127.27.254.62:46681 was elected leader, sending a full tablet report...
I20260808 15:59:52.989171 15544 ts_tablet_manager.cc:1428] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260808 15:59:52.989725 15542 raft_consensus.cc:348] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:52.989725 15543 raft_consensus.cc:348] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:52.990183 15542 raft_consensus.cc:374] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:52.990185 15543 raft_consensus.cc:374] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:52.990275 15542 raft_consensus.cc:729] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7886c34323a94bb7a850b3a838041d78, State: Initialized, Role: FOLLOWER
I20260808 15:59:52.990275 15543 raft_consensus.cc:729] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00238a78253a4ca9b4695d5784219952, State: Initialized, Role: FOLLOWER
I20260808 15:59:52.990846 15543 consensus_queue.cc:260] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:52.990846 15542 consensus_queue.cc:260] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:52.992626 15542 ts_tablet_manager.cc:1428] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260808 15:59:52.992761 15543 ts_tablet_manager.cc:1428] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260808 15:59:52.993072 15392 heartbeater.cc:503] Master 127.27.254.62:46681 was elected leader, sending a full tablet report...
I20260808 15:59:52.993717 15523 heartbeater.cc:503] Master 127.27.254.62:46681 was elected leader, sending a full tablet report...
W20260808 15:59:52.998229 15261 tablet.cc:2369] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:53.063092 15550 raft_consensus.cc:482] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:53.063453 15550 raft_consensus.cc:504] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:53.065078 15550 leader_election.cc:288] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00238a78253a4ca9b4695d5784219952 (127.27.254.3:33443), eb8a443cb553474db44704342dde82cc (127.27.254.1:45907)
I20260808 15:59:53.072507 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.065599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:40513 (local address 127.27.254.3:33443)
0808 15:59:53.065826 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:53.065836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:53.066295 (+ 459us) server_negotiation.cc:407] Connection header received
0808 15:59:53.066516 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:53.066524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:53.066616 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:53.066730 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:53.067400 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.068339 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.070558 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.071667 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.071791 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.071839 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:53.072214 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 15:59:53.072298 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:59:53.072711 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.065523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:33443 (local address 127.27.254.2:40513)
0808 15:59:53.066175 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.066200 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:53.066371 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:53.066776 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:53.066784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:53.066825 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:53.067242 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:53.067255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.068476 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.068481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:53.070373 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:53.070398 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.071864 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.071871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:53.072058 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.072093 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:59:53.072197 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:59:53.072333 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":84,"thread_start_us":370,"threads_started":1}
I20260808 15:59:53.073467 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.066303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.27.254.2:59079)
0808 15:59:53.066917 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.066933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:53.067032 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:53.067399 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:53.067404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:53.067427 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:53.067774 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:53.067784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.068948 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.068952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:53.071143 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:53.071158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.072758 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.072767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:53.072963 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.073004 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 15:59:53.073101 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 15:59:53.073190 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":297,"thread_start_us":314,"threads_started":1}
I20260808 15:59:53.073653 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.066444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:59079 (local address 127.27.254.1:45907)
0808 15:59:53.066618 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:53.066623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:53.067028 (+ 405us) server_negotiation.cc:407] Connection header received
0808 15:59:53.067137 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:53.067144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:53.067243 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:53.067349 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:53.067954 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.068809 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.071351 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.072370 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.072510 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.072571 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:53.073150 (+ 579us) server_negotiation.cc:299] Negotiation successful
0808 15:59:53.073286 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 15:59:53.073989 15477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79691e5ed51493cadd88d766da53ba3" candidate_uuid: "7886c34323a94bb7a850b3a838041d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00238a78253a4ca9b4695d5784219952" is_pre_election: true
I20260808 15:59:53.074597 15477 raft_consensus.cc:2454] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7886c34323a94bb7a850b3a838041d78 in term 0.
I20260808 15:59:53.074611 15215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79691e5ed51493cadd88d766da53ba3" candidate_uuid: "7886c34323a94bb7a850b3a838041d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb8a443cb553474db44704342dde82cc" is_pre_election: true
I20260808 15:59:53.075163 15215 raft_consensus.cc:2454] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7886c34323a94bb7a850b3a838041d78 in term 0.
I20260808 15:59:53.075350 15279 leader_election.cc:302] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00238a78253a4ca9b4695d5784219952, 7886c34323a94bb7a850b3a838041d78; no voters:
I20260808 15:59:53.075737 15550 raft_consensus.cc:2781] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:59:53.075888 15550 raft_consensus.cc:482] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:59:53.076001 15550 raft_consensus.cc:3037] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:53.078239 15550 raft_consensus.cc:504] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:53.078799 15550 leader_election.cc:288] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [CANDIDATE]: Term 1 election: Requested vote from peers 00238a78253a4ca9b4695d5784219952 (127.27.254.3:33443), eb8a443cb553474db44704342dde82cc (127.27.254.1:45907)
I20260808 15:59:53.079273 15215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79691e5ed51493cadd88d766da53ba3" candidate_uuid: "7886c34323a94bb7a850b3a838041d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb8a443cb553474db44704342dde82cc"
I20260808 15:59:53.079273 15477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79691e5ed51493cadd88d766da53ba3" candidate_uuid: "7886c34323a94bb7a850b3a838041d78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00238a78253a4ca9b4695d5784219952"
I20260808 15:59:53.079469 15215 raft_consensus.cc:3037] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:53.079501 15477 raft_consensus.cc:3037] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:53.082736 15477 raft_consensus.cc:2454] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7886c34323a94bb7a850b3a838041d78 in term 1.
I20260808 15:59:53.082736 15215 raft_consensus.cc:2454] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7886c34323a94bb7a850b3a838041d78 in term 1.
I20260808 15:59:53.083110 15279 leader_election.cc:302] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00238a78253a4ca9b4695d5784219952, 7886c34323a94bb7a850b3a838041d78; no voters:
I20260808 15:59:53.083367 15550 raft_consensus.cc:2781] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:53.083781 15550 raft_consensus.cc:686] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 1 LEADER]: Becoming Leader. State: Replica: 7886c34323a94bb7a850b3a838041d78, State: Running, Role: LEADER
I20260808 15:59:53.084226 15550 consensus_queue.cc:237] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } }
I20260808 15:59:53.088932 15072 catalog_manager.cc:5495] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7886c34323a94bb7a850b3a838041d78 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "7886c34323a94bb7a850b3a838041d78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7886c34323a94bb7a850b3a838041d78" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 46067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 } health_report { overall_health: UNKNOWN } } }
I20260808 15:59:53.112114 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.106911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.27.254.2:46067)
0808 15:59:53.107109 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:53.107113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:53.107150 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:59:53.107318 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:53.107322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:53.107445 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:53.107571 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:53.107946 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.108764 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.110150 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.110540 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.110627 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.111062 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:53.111518 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:59:53.111642 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:53.111770 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:59:53.111831 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:59:53.119812 15215 raft_consensus.cc:1261] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 1 FOLLOWER]: Refusing update from remote peer 7886c34323a94bb7a850b3a838041d78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:53.119848 15477 raft_consensus.cc:1261] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 1 FOLLOWER]: Refusing update from remote peer 7886c34323a94bb7a850b3a838041d78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:53.120762 15553 consensus_queue.cc:1035] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb8a443cb553474db44704342dde82cc" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 45907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:53.121101 15550 consensus_queue.cc:1035] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00238a78253a4ca9b4695d5784219952" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 33443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:53.132612 15557 mvcc.cc:205] Tried to move back new op lower bound from 7316294832609574912 to 7316294832476516352. Current Snapshot: MvccSnapshot[applied={T|T < 7316294832609574912}]
I20260808 15:59:53.146425 15072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:53.146710 15072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:53.150173 15072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a5f1ec8a694aa099bfc8790898df13: Sending DeleteTablet for 3 replicas of tablet d79691e5ed51493cadd88d766da53ba3
I20260808 15:59:53.151188 15326 tablet_service.cc:1514] Processing DeleteTablet for tablet d79691e5ed51493cadd88d766da53ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:53 UTC) from {username='slave'} at 127.0.0.1:50020
I20260808 15:59:53.151328 15195 tablet_service.cc:1514] Processing DeleteTablet for tablet d79691e5ed51493cadd88d766da53ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:53 UTC) from {username='slave'} at 127.0.0.1:59868
I20260808 15:59:53.151328 15457 tablet_service.cc:1514] Processing DeleteTablet for tablet d79691e5ed51493cadd88d766da53ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:53 UTC) from {username='slave'} at 127.0.0.1:36436
I20260808 15:59:53.153579 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15132
I20260808 15:59:53.159978 15566 tablet_replica.cc:330] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: stopping tablet replica
I20260808 15:59:53.160442 15566 raft_consensus.cc:2229] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:59:53.161201 15566 raft_consensus.cc:2258] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:53.161810 15567 tablet_replica.cc:330] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: stopping tablet replica
I20260808 15:59:53.162475 15567 raft_consensus.cc:2229] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:53.163046 15567 raft_consensus.cc:2258] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:53.163928 15568 tablet_replica.cc:330] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc: stopping tablet replica
I20260808 15:59:53.164177 15566 ts_tablet_manager.cc:1905] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:53.165256 15567 ts_tablet_manager.cc:1905] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:53.171083 15567 ts_tablet_manager.cc:1918] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:53.167876 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.154874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.27.254.1:45907)
0808 15:59:53.155091 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:53.155100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:53.159671 (+ 4571us) server_negotiation.cc:407] Connection header received
0808 15:59:53.160036 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:53.160044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:53.160171 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:53.160257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:53.160917 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.162148 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.164078 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.164734 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.164837 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.166318 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:53.166329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:53.166344 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:53.167009 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:53.167062 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:53.167086 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:53.167171 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:53.167414 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 15:59:53.167511 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:59:53.171254 15567 log.cc:1196] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/d79691e5ed51493cadd88d766da53ba3
I20260808 15:59:53.171959 15566 ts_tablet_manager.cc:1918] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:53.171967 15255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:53.172123 15566 log.cc:1196] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/d79691e5ed51493cadd88d766da53ba3
I20260808 15:59:53.172686 15566 ts_tablet_manager.cc:1939] T d79691e5ed51493cadd88d766da53ba3 P 7886c34323a94bb7a850b3a838041d78: Deleting consensus metadata
I20260808 15:59:53.173148 15567 ts_tablet_manager.cc:1939] T d79691e5ed51493cadd88d766da53ba3 P 00238a78253a4ca9b4695d5784219952: Deleting consensus metadata
I20260808 15:59:53.174475 15059 catalog_manager.cc:4841] TS 7886c34323a94bb7a850b3a838041d78 (127.27.254.2:46067): tablet d79691e5ed51493cadd88d766da53ba3 (table test-table [id=2e71b144f95c451f829c34e81a1d70b9]) successfully deleted
I20260808 15:59:53.175369 15057 catalog_manager.cc:4841] TS 00238a78253a4ca9b4695d5784219952 (127.27.254.3:33443): tablet d79691e5ed51493cadd88d766da53ba3 (table test-table [id=2e71b144f95c451f829c34e81a1d70b9]) successfully deleted
I20260808 15:59:53.171965 15568 raft_consensus.cc:2229] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:53.281334 15568 raft_consensus.cc:2258] T d79691e5ed51493cadd88d766da53ba3 P eb8a443cb553474db44704342dde82cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:53.281608 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15132
I20260808 15:59:53.299715 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15264
W20260808 15:59:53.299714 15057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:45907 (error 108)
I20260808 15:59:53.301877 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.301342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59880)
0808 15:59:53.301542 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.301707 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 15:59:53.302253 15057 catalog_manager.cc:4584] TS eb8a443cb553474db44704342dde82cc (127.27.254.1:45907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d79691e5ed51493cadd88d766da53ba3: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
I20260808 15:59:53.307299 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.301108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.27.254.2:46067)
0808 15:59:53.301290 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:53.301296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:53.301314 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:59:53.301378 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:53.301384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:53.301501 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:53.301611 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:53.302116 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.302857 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.304160 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.304669 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.304841 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.305923 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:53.305938 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:53.305952 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:53.306532 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:53.306583 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:53.306602 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:53.306713 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:53.306885 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 15:59:53.306979 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 15:59:53.307955 15387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:53.355930 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.355358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59894)
0808 15:59:53.355599 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.355719 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 15:59:53.357020 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.356623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59896)
0808 15:59:53.356782 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.356868 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 15:59:53.418154 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.417561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59904)
0808 15:59:53.417819 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.417941 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 15:59:53.418246 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15264
I20260808 15:59:53.419263 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.418877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59910)
0808 15:59:53.419045 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.419128 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:59:53.444940 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15395
I20260808 15:59:53.453720 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.445854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.27.254.3:33443)
0808 15:59:53.446068 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:53.446075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:53.446100 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:59:53.446261 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:53.446269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:53.446389 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:53.446514 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:53.446883 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.448307 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.449806 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:53.450638 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:53.450797 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:53.451855 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:53.451869 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:53.451891 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:53.452788 (+ 897us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:53.452855 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:53.452883 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:53.453008 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:53.453088 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 15:59:53.453229 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:59:53.454486 15518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:53.499812 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.499251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59918)
0808 15:59:53.499472 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.499619 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 15:59:53.500903 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.500526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59926)
0808 15:59:53.500729 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.500804 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 15:59:53.547335 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15395
I20260808 15:59:53.564772 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15041
I20260808 15:59:53.566157 15103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:53.674690 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:53.674083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:45907 (local address 127.0.0.1:59928)
0808 15:59:53.674308 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:53.674438 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 15:59:53.675009 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15041
2026-08-08T15:59:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2709 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:59:53 dist-test-slave-srgv krb5kdc[15577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:59:53 dist-test-slave-srgv krb5kdc[15577](info): set up 2 sockets
Aug 08 15:59:53 dist-test-slave-srgv krb5kdc[15577](info): commencing operation
krb5kdc: starting...
W20260808 15:59:54.787667 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 15:59:54 dist-test-slave-srgv krb5kdc[15577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 15:59:55.040362 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38531
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:38531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:55.365000 15590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:55.365285 15590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:55.365379 15590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:55.372479 15590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:55.372586 15590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:55.372634 15590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:55.372727 15590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:55.372798 15590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:55.382223 15590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:38531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:38531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:55.383708 15590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:55.384936 15590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:55.396631 15595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:55.396585 15598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:55.396548 15596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:55.397576 15590 server_base.cc:1034] running on GCE node
I20260808 15:59:55.398275 15590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:55.399499 15590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:55.400724 15590 hybrid_clock.cc:648] HybridClock initialized: now 1786204795400679 us; error 60 us; skew 500 ppm
I20260808 15:59:55.401044 15590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:55 dist-test-slave-srgv krb5kdc[15577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204795, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:55.409529 15590 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 15:59:55.411098 15590 webserver.cc:460] Webserver started at http://127.27.254.62:45419/ using document root <none> and password file <none>
I20260808 15:59:55.411809 15590 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:55.411957 15590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:55.412261 15590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:55.415347 15590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "675f4bb2990146b0b8d17cecdf73fbb3"
format_stamp: "Formatted at 2026-08-08 15:59:55 on dist-test-slave-srgv"
server_key: "03707c5fa8870a185117e82bc141882b"
server_key_iv: "b7d783f6764afae8ee15ed46e8ef3175"
server_key_version: "encryptionkey@0"
I20260808 15:59:55.416115 15590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "675f4bb2990146b0b8d17cecdf73fbb3"
format_stamp: "Formatted at 2026-08-08 15:59:55 on dist-test-slave-srgv"
server_key: "03707c5fa8870a185117e82bc141882b"
server_key_iv: "b7d783f6764afae8ee15ed46e8ef3175"
server_key_version: "encryptionkey@0"
I20260808 15:59:55.421285 15590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:59:55.424707 15605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:55.426116 15590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:59:55.426309 15590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "675f4bb2990146b0b8d17cecdf73fbb3"
format_stamp: "Formatted at 2026-08-08 15:59:55 on dist-test-slave-srgv"
server_key: "03707c5fa8870a185117e82bc141882b"
server_key_iv: "b7d783f6764afae8ee15ed46e8ef3175"
server_key_version: "encryptionkey@0"
I20260808 15:59:55.426508 15590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:55.443336 15590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:55.447619 15590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:55.447916 15590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:55.466216 15590 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:38531
I20260808 15:59:55.466238 15657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:38531 every 8 connection(s)
I20260808 15:59:55.467523 15590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:55.471045 15658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:55.474691 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15590
I20260808 15:59:55.474867 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:59:55.475451 28664 external_mini_cluster.cc:1428] Setting key 295a567582ad20327b3dc201eb6ba201
I20260808 15:59:55.484010 15658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3: Bootstrap starting.
I20260808 15:59:55.487751 15658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:55.489063 15658 log.cc:824] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:59:55 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:55.492285 15658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3: No bootstrap required, opened a new log
I20260808 15:59:55.500046 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:55.477530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.27.254.62:38531)
0808 15:59:55.478394 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:55.478405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:55.478444 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:59:55.480000 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:55.480034 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:55.480400 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:55.480723 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:55.482531 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:55.483771 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:55.485585 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:55.486272 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:55.486407 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:55.491426 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:55.491439 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:55.491461 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:55.491503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:55.495579 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:55.496278 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:55.496293 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:55.496302 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:55.496520 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:55.496906 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:55.496911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:55.496914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:55.497605 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:55.497936 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:55.498238 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 15:59:55.498673 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":402,"threads_started":1}
I20260808 15:59:55.502890 15658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38531 } }
I20260808 15:59:55.503250 15658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:55.503332 15658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675f4bb2990146b0b8d17cecdf73fbb3, State: Initialized, Role: FOLLOWER
I20260808 15:59:55.503770 15658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38531 } }
I20260808 15:59:55.503961 15658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:55.504055 15658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:55.504204 15658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:55.506656 15658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38531 } }
I20260808 15:59:55.507138 15658 leader_election.cc:302] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 675f4bb2990146b0b8d17cecdf73fbb3; no voters:
I20260808 15:59:55.507730 15658 leader_election.cc:288] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:59:55.507869 15663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:55.508576 15663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [term 1 LEADER]: Becoming Leader. State: Replica: 675f4bb2990146b0b8d17cecdf73fbb3, State: Running, Role: LEADER
I20260808 15:59:55.509096 15663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38531 } }
I20260808 15:59:55.509884 15658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:55.512159 15665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 675f4bb2990146b0b8d17cecdf73fbb3. Latest consensus state: current_term: 1 leader_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38531 } } }
I20260808 15:59:55.512612 15665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:55.512171 15664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675f4bb2990146b0b8d17cecdf73fbb3" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 38531 } } }
I20260808 15:59:55.513676 15664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:55.514771 15672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:55.519685 15672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:55.528509 15672 catalog_manager.cc:1347] Generated new cluster ID: 0409ce37cc124acca3e558c9e991eb9c
I20260808 15:59:55.528630 15672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:55.541662 15672 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:59:55.542884 15672 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:55.553251 15672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3: Generated new TSK 0
I20260808 15:59:55.554255 15672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
Entry for principal kudu/127.27.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab.
I20260808 15:59:55.615238 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38531
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:55.965898 15684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:55.966152 15684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:55.966302 15684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:55.973378 15684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:55.973521 15684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:55.973708 15684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:55.983197 15684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.1.keytab
--principal=kudu/127.27.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:55.984428 15684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:55.985769 15684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:55.998365 15690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:55.998409 15691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:55.998454 15693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:56.001243 15684 server_base.cc:1034] running on GCE node
I20260808 15:59:56.001988 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:56.002882 15684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:56.004073 15684 hybrid_clock.cc:648] HybridClock initialized: now 1786204796004012 us; error 64 us; skew 500 ppm
I20260808 15:59:56.004388 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:56 dist-test-slave-srgv krb5kdc[15577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204796, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:56.007539 15684 init.cc:377] Logged in from keytab as kudu/127.27.254.1@KRBTEST.COM (short username kudu)
I20260808 15:59:56.009023 15684 webserver.cc:460] Webserver started at http://127.27.254.1:35741/ using document root <none> and password file <none>
I20260808 15:59:56.009804 15684 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:56.009953 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:56.010258 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:56.013242 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "94ef7c7680834d43957e5d9a8835f1e3"
format_stamp: "Formatted at 2026-08-08 15:59:56 on dist-test-slave-srgv"
server_key: "52f18cf3e5ce5b3703cbb392211e3194"
server_key_iv: "f3f938d3072f00fbfe8aead641b8efe4"
server_key_version: "encryptionkey@0"
I20260808 15:59:56.014098 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "94ef7c7680834d43957e5d9a8835f1e3"
format_stamp: "Formatted at 2026-08-08 15:59:56 on dist-test-slave-srgv"
server_key: "52f18cf3e5ce5b3703cbb392211e3194"
server_key_iv: "f3f938d3072f00fbfe8aead641b8efe4"
server_key_version: "encryptionkey@0"
I20260808 15:59:56.019279 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:59:56.022594 15700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:56.023856 15684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:56.024032 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "94ef7c7680834d43957e5d9a8835f1e3"
format_stamp: "Formatted at 2026-08-08 15:59:56 on dist-test-slave-srgv"
server_key: "52f18cf3e5ce5b3703cbb392211e3194"
server_key_iv: "f3f938d3072f00fbfe8aead641b8efe4"
server_key_version: "encryptionkey@0"
I20260808 15:59:56.024228 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:56.055955 15684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:56.059854 15684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:56.060137 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:56.061067 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:56.062397 15684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:56.062496 15684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:56.062657 15684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:56.062731 15684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:56.095188 15684 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:33307
I20260808 15:59:56.095274 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:33307 every 8 connection(s)
I20260808 15:59:56.096443 15684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:56.097923 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15684
I20260808 15:59:56.098109 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:59:56.098678 28664 external_mini_cluster.cc:1428] Setting key 78dba6d9cfe4711d29e199b80b341bbe
Aug 08 15:59:56 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204796, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.1@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:56.118268 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:56.099008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:36311 (local address 127.27.254.62:38531)
0808 15:59:56.099556 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:56.099563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:56.100582 (+ 1019us) server_negotiation.cc:407] Connection header received
0808 15:59:56.102472 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:56.102477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:56.102557 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:56.102686 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:56.104118 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.104851 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.106429 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.106844 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.106978 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:56.110836 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:56.110845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:56.110853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:56.110898 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:56.114509 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:56.115388 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:56.115396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:56.115402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:56.115547 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:56.116199 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:56.116207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:56.116213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:56.116630 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:56.116822 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:56.117518 (+ 696us) server_negotiation.cc:299] Negotiation successful
0808 15:59:56.117783 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":319,"threads_started":1}
I20260808 15:59:56.118891 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:56.099668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38531 (local address 127.27.254.1:36311)
0808 15:59:56.100417 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:56.100469 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:56.102217 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:56.102922 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:56.102930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:56.103410 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:56.103954 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:56.103971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.105007 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.105012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:56.106280 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:56.106292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.107083 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.107090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:56.107325 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:56.108032 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:56.108060 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:56.110620 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:56.114701 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:56.114714 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:56.114725 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:56.115250 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:56.115668 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:56.115672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:56.115674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:56.116062 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:56.116789 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:56.116795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:56.117099 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 15:59:56.117325 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 15:59:56.117767 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":329,"threads_started":1}
I20260808 15:59:56.120466 15813 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38531
I20260808 15:59:56.120786 15813 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:56.121531 15813 heartbeater.cc:511] Master 127.27.254.62:38531 requested a full tablet report, sending...
I20260808 15:59:56.123584 15623 ts_manager.cc:194] Registered new tserver with Master: 94ef7c7680834d43957e5d9a8835f1e3 (127.27.254.1:33307)
I20260808 15:59:56.125036 15623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.1@KRBTEST.COM'} at 127.27.254.1:36311
WARNING: no policy specified for kudu/127.27.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
Entry for principal kudu/127.27.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab.
I20260808 15:59:56.156889 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38531
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:56.478816 15819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:56.479139 15819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:56.479308 15819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:56.486564 15819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:56.486722 15819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:56.486894 15819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:56.496578 15819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.2.keytab
--principal=kudu/127.27.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:56.497908 15819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:56.499280 15819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:56.512501 15827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:56.512538 15825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:56.512739 15824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:56.513372 15819 server_base.cc:1034] running on GCE node
I20260808 15:59:56.514019 15819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:56.515094 15819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:56.516328 15819 hybrid_clock.cc:648] HybridClock initialized: now 1786204796516268 us; error 66 us; skew 500 ppm
I20260808 15:59:56.516638 15819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:56 dist-test-slave-srgv krb5kdc[15577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204796, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:56.519750 15819 init.cc:377] Logged in from keytab as kudu/127.27.254.2@KRBTEST.COM (short username kudu)
I20260808 15:59:56.521133 15819 webserver.cc:460] Webserver started at http://127.27.254.2:42055/ using document root <none> and password file <none>
I20260808 15:59:56.521912 15819 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:56.522058 15819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:56.522361 15819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:56.525282 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "ec7adfd1314b49d4b186d18498d7f23c"
format_stamp: "Formatted at 2026-08-08 15:59:56 on dist-test-slave-srgv"
server_key: "d3df08ac80952a9958cb2d12dcca69bd"
server_key_iv: "2017dab7cd41ad6eae0db10014c8cdc7"
server_key_version: "encryptionkey@0"
I20260808 15:59:56.526101 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "ec7adfd1314b49d4b186d18498d7f23c"
format_stamp: "Formatted at 2026-08-08 15:59:56 on dist-test-slave-srgv"
server_key: "d3df08ac80952a9958cb2d12dcca69bd"
server_key_iv: "2017dab7cd41ad6eae0db10014c8cdc7"
server_key_version: "encryptionkey@0"
I20260808 15:59:56.531347 15819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:59:56.534605 15835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:56.535847 15819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:56.536018 15819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "ec7adfd1314b49d4b186d18498d7f23c"
format_stamp: "Formatted at 2026-08-08 15:59:56 on dist-test-slave-srgv"
server_key: "d3df08ac80952a9958cb2d12dcca69bd"
server_key_iv: "2017dab7cd41ad6eae0db10014c8cdc7"
server_key_version: "encryptionkey@0"
I20260808 15:59:56.536231 15819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:56.564791 15819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:56.569048 15819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:56.569360 15819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:56.570439 15819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:56.571790 15819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:56.571862 15819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:56.572011 15819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:56.572085 15819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:56.604960 15819 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:35155
I20260808 15:59:56.605062 15947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:35155 every 8 connection(s)
I20260808 15:59:56.606372 15819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:56.611256 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15819
I20260808 15:59:56.611452 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:59:56.611900 28664 external_mini_cluster.cc:1428] Setting key f9f52286aabf00b372e10738f6e04397
Aug 08 15:59:56 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204796, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.2@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:56.627173 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:56.608984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:47561 (local address 127.27.254.62:38531)
0808 15:59:56.609173 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:56.609177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:56.610505 (+ 1328us) server_negotiation.cc:407] Connection header received
0808 15:59:56.612674 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:56.612680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:56.612776 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:56.612897 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:56.614601 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.615472 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.617083 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.617605 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.617742 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:56.621287 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:56.621292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:56.621295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:56.621327 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:56.623966 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:56.624875 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:56.624880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:56.624882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:56.625008 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:56.625497 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:56.625502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:56.625505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:56.625776 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:56.625894 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:56.626767 (+ 873us) server_negotiation.cc:299] Negotiation successful
0808 15:59:56.626937 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:59:56.628785 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:56.609568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38531 (local address 127.27.254.2:47561)
0808 15:59:56.610347 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:56.610389 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:56.612390 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:56.613132 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:56.613141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:56.613736 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:56.614422 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:56.614440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.615626 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.615630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:56.616940 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:56.616952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:56.617835 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:56.617842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:56.618066 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:56.618729 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:56.618751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:56.621062 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:56.624116 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:56.624127 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:56.624146 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:56.624750 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:56.625123 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:56.625131 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:56.625137 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:56.625377 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:56.625907 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:56.625917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:56.626363 (+ 446us) client_negotiation.cc:769] Sending connection context
0808 15:59:56.626682 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 15:59:56.627262 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":273,"threads_started":1}
I20260808 15:59:56.630329 15948 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38531
I20260808 15:59:56.630659 15948 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:56.631453 15948 heartbeater.cc:511] Master 127.27.254.62:38531 requested a full tablet report, sending...
I20260808 15:59:56.632908 15623 ts_manager.cc:194] Registered new tserver with Master: ec7adfd1314b49d4b186d18498d7f23c (127.27.254.2:35155)
I20260808 15:59:56.633793 15623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.2@KRBTEST.COM'} at 127.27.254.2:47561
WARNING: no policy specified for kudu/127.27.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
Entry for principal kudu/127.27.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab.
I20260808 15:59:56.670859 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38531
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:59:56.995905 15953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:56.996155 15953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:56.996246 15953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:57.003057 15953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:57.003157 15953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:57.003268 15953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:57.012776 15953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.3.keytab
--principal=kudu/127.27.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:38531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:57.014065 15953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:57.015362 15953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:57.028106 15958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:57.028165 15961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:57.028204 15959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:57.029506 15953 server_base.cc:1034] running on GCE node
I20260808 15:59:57.030158 15953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:57.031162 15953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:57.032398 15953 hybrid_clock.cc:648] HybridClock initialized: now 1786204797032337 us; error 44 us; skew 500 ppm
I20260808 15:59:57.032709 15953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:59:57 dist-test-slave-srgv krb5kdc[15577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:59:57.035902 15953 init.cc:377] Logged in from keytab as kudu/127.27.254.3@KRBTEST.COM (short username kudu)
I20260808 15:59:57.037293 15953 webserver.cc:460] Webserver started at http://127.27.254.3:40827/ using document root <none> and password file <none>
I20260808 15:59:57.037974 15953 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:57.038105 15953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:57.038411 15953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:57.041257 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "6d3157f1126440c8aac9f29c0abbca9e"
format_stamp: "Formatted at 2026-08-08 15:59:57 on dist-test-slave-srgv"
server_key: "2ed0c17387d96812957424a40a417dc9"
server_key_iv: "f4a9afef107ea38d55d8bf53263c40c1"
server_key_version: "encryptionkey@0"
I20260808 15:59:57.042065 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "6d3157f1126440c8aac9f29c0abbca9e"
format_stamp: "Formatted at 2026-08-08 15:59:57 on dist-test-slave-srgv"
server_key: "2ed0c17387d96812957424a40a417dc9"
server_key_iv: "f4a9afef107ea38d55d8bf53263c40c1"
server_key_version: "encryptionkey@0"
I20260808 15:59:57.046839 15953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:59:57.050138 15968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:57.051395 15953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:59:57.051560 15953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "6d3157f1126440c8aac9f29c0abbca9e"
format_stamp: "Formatted at 2026-08-08 15:59:57 on dist-test-slave-srgv"
server_key: "2ed0c17387d96812957424a40a417dc9"
server_key_iv: "f4a9afef107ea38d55d8bf53263c40c1"
server_key_version: "encryptionkey@0"
I20260808 15:59:57.051743 15953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:57.064780 15953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:57.068843 15953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:57.069132 15953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:57.070055 15953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:57.071288 15953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:57.071360 15953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:57.071502 15953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:57.071565 15953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:57.104667 15953 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:38341
I20260808 15:59:57.104759 16081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:38341 every 8 connection(s)
I20260808 15:59:57.105952 15953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:57.115144 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 15953
I20260808 15:59:57.115386 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:59:57.115803 28664 external_mini_cluster.cc:1428] Setting key 04faeb59adf34238bf5e0e8e206b57e3
Aug 08 15:59:57 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.3@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 15:59:57.125694 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.108514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:42719 (local address 127.27.254.62:38531)
0808 15:59:57.108729 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.108733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.109885 (+ 1152us) server_negotiation.cc:407] Connection header received
0808 15:59:57.111727 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.111733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.111845 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.111958 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:57.113440 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.114316 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.116090 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.116650 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.116801 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.120100 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:57.120105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:57.120108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:57.120139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:57.122803 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.123487 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.123494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.123498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.123629 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.124060 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.124065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.124068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.124331 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:57.124491 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.125118 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.125326 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:59:57.126593 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.108997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:38531 (local address 127.27.254.3:42719)
0808 15:59:57.109728 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.109772 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:57.111465 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:57.112183 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:57.112191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:57.112669 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:57.113240 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.113264 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.114459 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.114471 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.115955 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.115968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.116869 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.116879 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.117134 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.117843 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:59:57.117864 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:57.119896 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:57.122953 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:57.122961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:57.122975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:57.123363 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:57.123750 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:59:57.123754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:59:57.123758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:59:57.123944 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:59:57.124460 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:57.124468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:57.124772 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 15:59:57.125020 (+ 248us) client_negotiation.cc:240] Negotiation successful
0808 15:59:57.125539 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":272,"threads_started":1}
I20260808 15:59:57.127897 16082 heartbeater.cc:347] Connected to a master server at 127.27.254.62:38531
I20260808 15:59:57.127975 15813 heartbeater.cc:503] Master 127.27.254.62:38531 was elected leader, sending a full tablet report...
I20260808 15:59:57.128207 16082 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:57.129078 16082 heartbeater.cc:511] Master 127.27.254.62:38531 requested a full tablet report, sending...
I20260808 15:59:57.130848 15623 ts_manager.cc:194] Registered new tserver with Master: 6d3157f1126440c8aac9f29c0abbca9e (127.27.254.3:38341)
I20260808 15:59:57.131633 15623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.254.3@KRBTEST.COM'} at 127.27.254.3:42719
I20260808 15:59:57.143879 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:59:57.160022 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.147890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.27.254.62:38531)
0808 15:59:57.148101 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.148110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.148509 (+ 399us) server_negotiation.cc:407] Connection header received
0808 15:59:57.148636 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.148639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.148695 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.148789 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:57.150250 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.150938 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.152169 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.152546 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.152635 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.154940 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:57.154945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:57.154948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:57.154976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:57.157707 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.158325 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.158329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.158331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.158450 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.158761 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.158764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.158766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.159056 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:57.159243 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.159524 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.159681 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:59:57.163489 15623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:59:57.170006 15623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:59:57.191180 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.181074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43404)
0808 15:59:57.182133 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.182175 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:57.182341 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:57.183302 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:57.183309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:57.183338 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:57.183701 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.183715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.185802 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.185807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.188239 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.188253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.190165 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.190172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.190431 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.190483 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 15:59:57.190620 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 15:59:57.190771 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":30,"thread_start_us":690,"threads_started":1}
I20260808 15:59:57.191455 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.183049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.27.254.3:38341)
0808 15:59:57.184068 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.184079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.184113 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:59:57.184728 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.184745 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.184987 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.185164 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:57.185858 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.187090 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.189202 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.190270 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.190396 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.190604 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.190971 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.191135 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":700,"threads_started":1}
I20260808 15:59:57.191749 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.181402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.27.254.1:33307)
0808 15:59:57.182677 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.182683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.182705 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:59:57.182778 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.182783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.183047 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.183323 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:57.183905 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.185635 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.188460 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.189950 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.190077 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.190298 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.191282 (+ 984us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.191444 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":972,"threads_started":1}
I20260808 15:59:57.192248 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.182516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:38341 (local address 127.0.0.1:60902)
0808 15:59:57.182982 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.183000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:57.183107 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:57.185257 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:57.185262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:57.185288 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:57.185672 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.185687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.187224 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.187232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.189053 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.189064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.190612 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.190619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.190774 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.190811 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:59:57.190909 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:59:57.191038 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3,"thread_start_us":352,"threads_started":1}
I20260808 15:59:57.192991 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.183174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35155 (local address 127.0.0.1:60782)
0808 15:59:57.184777 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.184793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:57.184910 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:57.185569 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:57.185574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:57.185598 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:57.186118 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.186146 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.188115 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.188121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.190342 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.190356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.192178 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.192187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.192396 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.192436 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:59:57.192532 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 15:59:57.192653 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"thread_start_us":1313,"threads_started":1}
I20260808 15:59:57.193820 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.183676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.27.254.2:35155)
0808 15:59:57.184338 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.184348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.185026 (+ 678us) server_negotiation.cc:407] Connection header received
0808 15:59:57.185133 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.185139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.185335 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.185555 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:57.186339 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.187925 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.190499 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.191952 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.192099 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.192289 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.193467 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.193600 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":395,"threads_started":1}
I20260808 15:59:57.195914 16017 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc9dd13c72a4b85b19392771151c27c (DEFAULT_TABLE table=test-table [id=46338485de4d401e89487185b856a9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:57.195915 15748 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc9dd13c72a4b85b19392771151c27c (DEFAULT_TABLE table=test-table [id=46338485de4d401e89487185b856a9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:57.196533 15883 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc9dd13c72a4b85b19392771151c27c (DEFAULT_TABLE table=test-table [id=46338485de4d401e89487185b856a9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:59:57.197422 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc9dd13c72a4b85b19392771151c27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:57.197435 16017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc9dd13c72a4b85b19392771151c27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:57.197906 15883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc9dd13c72a4b85b19392771151c27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:57.208400 16100 tablet_bootstrap.cc:492] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Bootstrap starting.
I20260808 15:59:57.212164 16102 tablet_bootstrap.cc:492] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Bootstrap starting.
I20260808 15:59:57.212572 16100 tablet_bootstrap.cc:654] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:57.214010 16100 log.cc:824] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:57.215476 16101 tablet_bootstrap.cc:492] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Bootstrap starting.
I20260808 15:59:57.215826 16102 tablet_bootstrap.cc:654] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:57.217111 16102 log.cc:824] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:57.217381 16100 tablet_bootstrap.cc:492] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: No bootstrap required, opened a new log
I20260808 15:59:57.217710 16100 ts_tablet_manager.cc:1397] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 15:59:57.218703 16101 tablet_bootstrap.cc:654] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:57.219867 16101 log.cc:824] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:57.220188 16102 tablet_bootstrap.cc:492] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: No bootstrap required, opened a new log
I20260808 15:59:57.220481 16102 ts_tablet_manager.cc:1397] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:59:57.222750 16101 tablet_bootstrap.cc:492] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: No bootstrap required, opened a new log
I20260808 15:59:57.222993 16101 ts_tablet_manager.cc:1397] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:59:57.226204 16100 raft_consensus.cc:348] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.226531 16100 raft_consensus.cc:374] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:57.226631 16100 raft_consensus.cc:729] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec7adfd1314b49d4b186d18498d7f23c, State: Initialized, Role: FOLLOWER
I20260808 15:59:57.227232 16100 consensus_queue.cc:260] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.228279 15948 heartbeater.cc:503] Master 127.27.254.62:38531 was elected leader, sending a full tablet report...
I20260808 15:59:57.228912 16100 ts_tablet_manager.cc:1428] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260808 15:59:57.231042 16102 raft_consensus.cc:348] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.231482 16102 raft_consensus.cc:374] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:57.231573 16102 raft_consensus.cc:729] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3157f1126440c8aac9f29c0abbca9e, State: Initialized, Role: FOLLOWER
I20260808 15:59:57.232152 16102 consensus_queue.cc:260] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.233191 16082 heartbeater.cc:503] Master 127.27.254.62:38531 was elected leader, sending a full tablet report...
I20260808 15:59:57.233246 16101 raft_consensus.cc:348] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.233707 16101 raft_consensus.cc:374] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:57.233825 16101 raft_consensus.cc:729] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef7c7680834d43957e5d9a8835f1e3, State: Initialized, Role: FOLLOWER
I20260808 15:59:57.233953 16102 ts_tablet_manager.cc:1428] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260808 15:59:57.234395 16101 consensus_queue.cc:260] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.236001 16101 ts_tablet_manager.cc:1428] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260808 15:59:57.348918 15814 tablet.cc:2369] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:57.352770 16107 raft_consensus.cc:482] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:59:57.353057 16107 raft_consensus.cc:504] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.354719 16107 leader_election.cc:288] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ef7c7680834d43957e5d9a8835f1e3 (127.27.254.1:33307), ec7adfd1314b49d4b186d18498d7f23c (127.27.254.2:35155)
W20260808 15:59:57.357474 16083 tablet.cc:2369] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:59:57.358355 15949 tablet.cc:2369] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:59:57.361001 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.354927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.27.254.3:46545)
0808 15:59:57.355167 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.355194 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:57.355348 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:57.355832 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:57.355837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:57.355869 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:57.356095 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.356108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.357318 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.357325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.359212 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.359224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.360283 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.360289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.360467 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.360490 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:59:57.360595 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 15:59:57.360720 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 15:59:57.360991 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.355019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:46545 (local address 127.27.254.1:33307)
0808 15:59:57.355221 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.355229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.355250 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:59:57.355536 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.355545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.355645 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.355792 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:57.356238 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.357193 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.359369 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.360119 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.360283 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.360354 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.360619 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.360740 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:59:57.361626 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.355423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:48587 (local address 127.27.254.2:35155)
0808 15:59:57.355641 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.355648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.356206 (+ 558us) server_negotiation.cc:407] Connection header received
0808 15:59:57.356363 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.356368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.356443 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.356532 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:59:57.357041 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.358138 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.359873 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.360640 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.360830 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.360899 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.361260 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.361385 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:59:57.361984 15768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc9dd13c72a4b85b19392771151c27c" candidate_uuid: "6d3157f1126440c8aac9f29c0abbca9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef7c7680834d43957e5d9a8835f1e3" is_pre_election: true
I20260808 15:59:57.362176 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.355291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:35155 (local address 127.27.254.3:48587)
0808 15:59:57.356093 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.356109 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:57.356228 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:57.356548 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:57.356551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:57.356569 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:59:57.356884 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.356891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.358329 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.358333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.359714 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:57.359728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.360843 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.360849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:57.361033 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.361099 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 15:59:57.361815 (+ 716us) client_negotiation.cc:240] Negotiation successful
0808 15:59:57.361955 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":402,"threads_started":1}
I20260808 15:59:57.362478 15768 raft_consensus.cc:2454] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3157f1126440c8aac9f29c0abbca9e in term 0.
I20260808 15:59:57.363134 15971 leader_election.cc:302] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3157f1126440c8aac9f29c0abbca9e, 94ef7c7680834d43957e5d9a8835f1e3; no voters:
I20260808 15:59:57.363200 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc9dd13c72a4b85b19392771151c27c" candidate_uuid: "6d3157f1126440c8aac9f29c0abbca9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec7adfd1314b49d4b186d18498d7f23c" is_pre_election: true
I20260808 15:59:57.363559 16107 raft_consensus.cc:2781] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:59:57.363703 16107 raft_consensus.cc:482] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:59:57.363709 15903 raft_consensus.cc:2454] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3157f1126440c8aac9f29c0abbca9e in term 0.
I20260808 15:59:57.363821 16107 raft_consensus.cc:3037] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:57.366092 16107 raft_consensus.cc:504] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.366662 16107 leader_election.cc:288] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [CANDIDATE]: Term 1 election: Requested vote from peers 94ef7c7680834d43957e5d9a8835f1e3 (127.27.254.1:33307), ec7adfd1314b49d4b186d18498d7f23c (127.27.254.2:35155)
I20260808 15:59:57.367050 15768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc9dd13c72a4b85b19392771151c27c" candidate_uuid: "6d3157f1126440c8aac9f29c0abbca9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef7c7680834d43957e5d9a8835f1e3"
I20260808 15:59:57.367087 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc9dd13c72a4b85b19392771151c27c" candidate_uuid: "6d3157f1126440c8aac9f29c0abbca9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec7adfd1314b49d4b186d18498d7f23c"
I20260808 15:59:57.367244 15768 raft_consensus.cc:3037] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:57.367271 15903 raft_consensus.cc:3037] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:57.369861 15768 raft_consensus.cc:2454] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3157f1126440c8aac9f29c0abbca9e in term 1.
I20260808 15:59:57.369868 15903 raft_consensus.cc:2454] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3157f1126440c8aac9f29c0abbca9e in term 1.
I20260808 15:59:57.370244 15971 leader_election.cc:302] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3157f1126440c8aac9f29c0abbca9e, 94ef7c7680834d43957e5d9a8835f1e3; no voters:
I20260808 15:59:57.370517 16107 raft_consensus.cc:2781] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:57.370955 16107 raft_consensus.cc:686] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 1 LEADER]: Becoming Leader. State: Replica: 6d3157f1126440c8aac9f29c0abbca9e, State: Running, Role: LEADER
I20260808 15:59:57.371457 16107 consensus_queue.cc:237] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } }
I20260808 15:59:57.375994 15622 catalog_manager.cc:5495] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d3157f1126440c8aac9f29c0abbca9e (127.27.254.3). New cstate: current_term: 1 leader_uuid: "6d3157f1126440c8aac9f29c0abbca9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3157f1126440c8aac9f29c0abbca9e" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 } health_report { overall_health: UNKNOWN } } }
I20260808 15:59:57.426981 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.421658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.27.254.3:38341)
0808 15:59:57.421900 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.421905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.421920 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:59:57.422011 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.422014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.422098 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.422255 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:59:57.422660 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.423420 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.424837 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.425236 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.425371 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.425908 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:59:57.426399 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:59:57.426561 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.426628 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.426708 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:59:57.435005 15903 raft_consensus.cc:1261] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 1 FOLLOWER]: Refusing update from remote peer 6d3157f1126440c8aac9f29c0abbca9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:57.435036 15768 raft_consensus.cc:1261] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 1 FOLLOWER]: Refusing update from remote peer 6d3157f1126440c8aac9f29c0abbca9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:59:57.435935 16111 consensus_queue.cc:1035] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef7c7680834d43957e5d9a8835f1e3" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:57.436306 16107 consensus_queue.cc:1035] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7adfd1314b49d4b186d18498d7f23c" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:59:57.449038 16114 mvcc.cc:205] Tried to move back new op lower bound from 7316294850284630016 to 7316294850037186560. Current Snapshot: MvccSnapshot[applied={T|T < 7316294850284630016}]
I20260808 15:59:57.461236 15623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:57.461592 15623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:59:57.465088 15623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 675f4bb2990146b0b8d17cecdf73fbb3: Sending DeleteTablet for 3 replicas of tablet 9fc9dd13c72a4b85b19392771151c27c
I20260808 15:59:57.466554 15748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc9dd13c72a4b85b19392771151c27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:57 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:43404
I20260808 15:59:57.466737 15883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc9dd13c72a4b85b19392771151c27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:57 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:60782
I20260808 15:59:57.466722 16017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc9dd13c72a4b85b19392771151c27c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:59:57 UTC) from {username='kudu', principal='kudu/127.27.254.62@KRBTEST.COM'} at 127.0.0.1:60902
I20260808 15:59:57.470726 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15684
I20260808 15:59:57.479497 16125 tablet_replica.cc:330] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: stopping tablet replica
I20260808 15:59:57.480038 16126 tablet_replica.cc:330] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: stopping tablet replica
I20260808 15:59:57.481033 16126 raft_consensus.cc:2229] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:57.481709 16125 raft_consensus.cc:2229] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:59:57.482630 16126 raft_consensus.cc:2258] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:57.483323 16125 raft_consensus.cc:2258] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:57.486162 16126 ts_tablet_manager.cc:1905] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:57.491096 16125 ts_tablet_manager.cc:1905] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:57.492941 16124 tablet_replica.cc:330] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: stopping tablet replica
I20260808 15:59:57.493348 16124 raft_consensus.cc:2229] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:59:57.493852 16124 raft_consensus.cc:2258] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:59:57.496132 16126 ts_tablet_manager.cc:1918] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:57.496338 16126 log.cc:1196] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/9fc9dd13c72a4b85b19392771151c27c
I20260808 15:59:57.496898 16126 ts_tablet_manager.cc:1939] T 9fc9dd13c72a4b85b19392771151c27c P ec7adfd1314b49d4b186d18498d7f23c: Deleting consensus metadata
I20260808 15:59:57.498684 15607 catalog_manager.cc:4841] TS ec7adfd1314b49d4b186d18498d7f23c (127.27.254.2:35155): tablet 9fc9dd13c72a4b85b19392771151c27c (table test-table [id=46338485de4d401e89487185b856a9e3]) successfully deleted
Aug 08 15:59:57 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.1@KRBTEST.COM
I20260808 15:59:57.500970 16124 ts_tablet_manager.cc:1905] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:59:57.505371 16125 ts_tablet_manager.cc:1918] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:57.505563 16125 log.cc:1196] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/9fc9dd13c72a4b85b19392771151c27c
I20260808 15:59:57.506117 16125 ts_tablet_manager.cc:1939] T 9fc9dd13c72a4b85b19392771151c27c P 6d3157f1126440c8aac9f29c0abbca9e: Deleting consensus metadata
I20260808 15:59:57.507839 15607 catalog_manager.cc:4841] TS 6d3157f1126440c8aac9f29c0abbca9e (127.27.254.3:38341): tablet 9fc9dd13c72a4b85b19392771151c27c (table test-table [id=46338485de4d401e89487185b856a9e3]) successfully deleted
I20260808 15:59:57.508222 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.472427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.27.254.1:33307)
0808 15:59:57.472646 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.472655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.487066 (+ 14411us) server_negotiation.cc:407] Connection header received
0808 15:59:57.487513 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.487521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.487630 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.487705 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:57.489571 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.490726 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.494251 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.494883 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.494984 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.500599 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:57.500611 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:57.500627 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:57.500666 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:57.504552 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.505559 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.505569 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.505572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.505742 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.506209 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.506214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.506220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.506717 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:57.507011 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.507613 (+ 602us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.507851 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:59:57.509122 16124 ts_tablet_manager.cc:1918] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:59:57.509302 16124 log.cc:1196] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/9fc9dd13c72a4b85b19392771151c27c
I20260808 15:59:57.509696 15808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:57.637660 16124 ts_tablet_manager.cc:1939] T 9fc9dd13c72a4b85b19392771151c27c P 94ef7c7680834d43957e5d9a8835f1e3: Deleting consensus metadata
I20260808 15:59:57.638336 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15684
I20260808 15:59:57.655718 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15819
W20260808 15:59:57.655789 15607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.254.1:33307 (error 108)
I20260808 15:59:57.657871 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.657387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43430)
0808 15:59:57.657651 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.657738 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260808 15:59:57.658134 15607 catalog_manager.cc:4584] TS 94ef7c7680834d43957e5d9a8835f1e3 (127.27.254.1:33307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fc9dd13c72a4b85b19392771151c27c: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Aug 08 15:59:57 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.2@KRBTEST.COM
I20260808 15:59:57.672749 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.656678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.27.254.2:35155)
0808 15:59:57.656864 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.656872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.656938 (+ 66us) server_negotiation.cc:407] Connection header received
0808 15:59:57.657165 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.657171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.657280 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.657391 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:57.659109 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.659804 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.661644 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.662043 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.662136 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.667000 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:57.667036 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:57.667049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:57.667081 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:57.670055 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.670698 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.670705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.670707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.670843 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.671209 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.671214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.671216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.671569 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:57.671858 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.672187 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.672417 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:59:57.673432 15943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:57.711694 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.711194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43442)
0808 15:59:57.711410 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.711523 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 15:59:57.712848 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.712445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43456)
0808 15:59:57.712611 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.712695 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:59:57.762473 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15819
I20260808 15:59:57.773738 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.773079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43470)
0808 15:59:57.773366 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.773513 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 15:59:57.774981 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.774557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43472)
0808 15:59:57.774748 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.774840 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:59:57.781298 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15953
Aug 08 15:59:57 dist-test-slave-srgv krb5kdc[15577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.3@KRBTEST.COM
I20260808 15:59:57.799345 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.782237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.27.254.3:38341)
0808 15:59:57.782421 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:57.782434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:57.782458 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:59:57.782535 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:57.782539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:57.782598 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:57.782716 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:57.784011 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.784953 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.786962 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:57.787428 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:57.787528 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:57.792935 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:57.792947 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:57.792962 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:59:57.793002 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:57.796286 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.796999 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.797007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.797009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.797169 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:59:57.797738 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:59:57.797741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:59:57.797743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:59:57.798088 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:57.798415 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:57.798719 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 15:59:57.798943 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:59:57.800058 16077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:57.855739 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.855163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43478)
0808 15:59:57.855417 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.855531 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 15:59:57.856879 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:57.856476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43482)
0808 15:59:57.856679 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:57.856760 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 15:59:57.922206 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15953
I20260808 15:59:57.942963 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 15590
I20260808 15:59:57.944355 15653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:59:58.057476 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:58.056647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33307 (local address 127.0.0.1:43492)
0808 15:59:58.056977 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:58.057100 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.1:33307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260808 15:59:58.057600 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 15590
2026-08-08T15:59:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4383 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T15:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:59:58Z Disabled control of system clock
I20260808 15:59:58.135842 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33315
--webserver_interface=127.27.254.62
--webserver_port=0
--builtin_ntp_servers=127.27.254.20:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:59:58.456784 16136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:58.457051 16136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:58.457193 16136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:58.464424 16136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:59:58.464526 16136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:58.464620 16136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:58.464730 16136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:59:58.464809 16136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:59:58.473733 16136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:58.474941 16136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:58.476680 16136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:58.488019 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:58.488118 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:58.488101 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:58.488710 16136 server_base.cc:1034] running on GCE node
I20260808 15:59:58.489511 16136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:58.490800 16136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:58.492110 16136 hybrid_clock.cc:648] HybridClock initialized: now 1786204798492070 us; error 29 us; skew 500 ppm
I20260808 15:59:58.492491 16136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:58.494797 16136 webserver.cc:460] Webserver started at http://127.27.254.62:45245/ using document root <none> and password file <none>
I20260808 15:59:58.495486 16136 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:58.495597 16136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:58.495975 16136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:58.498808 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "976f73db29074b61a498ae9528e5b00d"
format_stamp: "Formatted at 2026-08-08 15:59:58 on dist-test-slave-srgv"
server_key: "e236508d33e53d8ca836f09b9aeca2d6"
server_key_iv: "765e0c4a5c3a313ffbd489a3923a6dc4"
server_key_version: "encryptionkey@0"
I20260808 15:59:58.499621 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "976f73db29074b61a498ae9528e5b00d"
format_stamp: "Formatted at 2026-08-08 15:59:58 on dist-test-slave-srgv"
server_key: "e236508d33e53d8ca836f09b9aeca2d6"
server_key_iv: "765e0c4a5c3a313ffbd489a3923a6dc4"
server_key_version: "encryptionkey@0"
I20260808 15:59:58.504418 16136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 15:59:58.507573 16150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:58.508857 16136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 15:59:58.509044 16136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "976f73db29074b61a498ae9528e5b00d"
format_stamp: "Formatted at 2026-08-08 15:59:58 on dist-test-slave-srgv"
server_key: "e236508d33e53d8ca836f09b9aeca2d6"
server_key_iv: "765e0c4a5c3a313ffbd489a3923a6dc4"
server_key_version: "encryptionkey@0"
I20260808 15:59:58.509250 16136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:58.519991 16136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:58.520643 16136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:58.520877 16136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:58.538635 16136 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33315
I20260808 15:59:58.538669 16201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33315 every 8 connection(s)
I20260808 15:59:58.539921 16136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 15:59:58.543515 16202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:59:58.547248 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16136
I20260808 15:59:58.547442 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 15:59:58.547999 28664 external_mini_cluster.cc:1428] Setting key c81c7aa719cf17a6821cdab1b0c688fc
I20260808 15:59:58.555804 16202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d: Bootstrap starting.
I20260808 15:59:58.558753 16202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d: Neither blocks nor log segments found. Creating new log.
I20260808 15:59:58.559806 16202 log.cc:824] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:59:58.560885 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:58.549908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.27.254.62:33315)
0808 15:59:58.550900 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:58.550911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:58.550949 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:59:58.552466 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:58.552486 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:58.552927 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:58.553321 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:58.553969 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:58.555163 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:58.556566 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:58.557233 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:58.557452 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:58.558608 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:58.558615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:58.558628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:58.559206 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:58.559285 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:58.559305 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:58.559428 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:58.559659 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 15:59:58.559972 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":417,"threads_started":1}
I20260808 15:59:58.562665 16202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d: No bootstrap required, opened a new log
I20260808 15:59:58.570346 16202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f73db29074b61a498ae9528e5b00d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33315 } }
I20260808 15:59:58.570693 16202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:59:58.570804 16202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976f73db29074b61a498ae9528e5b00d, State: Initialized, Role: FOLLOWER
I20260808 15:59:58.571332 16202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f73db29074b61a498ae9528e5b00d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33315 } }
I20260808 15:59:58.571578 16202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:59:58.571712 16202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:59:58.571887 16202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:59:58.574218 16202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f73db29074b61a498ae9528e5b00d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33315 } }
I20260808 15:59:58.574626 16202 leader_election.cc:302] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 976f73db29074b61a498ae9528e5b00d; no voters:
I20260808 15:59:58.575107 16202 leader_election.cc:288] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:59:58.575248 16207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:59:58.575932 16207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [term 1 LEADER]: Becoming Leader. State: Replica: 976f73db29074b61a498ae9528e5b00d, State: Running, Role: LEADER
I20260808 15:59:58.576429 16207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f73db29074b61a498ae9528e5b00d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33315 } }
I20260808 15:59:58.577088 16202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:59:58.579217 16208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "976f73db29074b61a498ae9528e5b00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f73db29074b61a498ae9528e5b00d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33315 } } }
I20260808 15:59:58.579715 16208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:58.579332 16209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976f73db29074b61a498ae9528e5b00d. Latest consensus state: current_term: 1 leader_uuid: "976f73db29074b61a498ae9528e5b00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f73db29074b61a498ae9528e5b00d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33315 } } }
I20260808 15:59:58.580783 16209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d [sys.catalog]: This master's current role is: LEADER
I20260808 15:59:58.581367 16215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:59:58.586495 16215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:59:58.594820 16215 catalog_manager.cc:1347] Generated new cluster ID: d7630a75e5624d5ba2e7bb214b91f1c3
I20260808 15:59:58.594957 16215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:59:58.603817 16215 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:59:58.605445 16215 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:59:58.613976 16215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d: Generated new TSK 0
I20260808 15:59:58.614703 16215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:59:58.624830 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.1:0
--local_ip_for_outbound_sockets=127.27.254.1
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33315
--builtin_ntp_servers=127.27.254.20:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:59:58.954187 16226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:58.954500 16226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:58.954658 16226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:58.961570 16226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:58.961695 16226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:58.961897 16226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.1
I20260808 15:59:58.970582 16226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.254.1
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.1
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:58.971822 16226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:58.973645 16226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:58.986543 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:58.986505 16234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:58.986733 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:58.987679 16226 server_base.cc:1034] running on GCE node
I20260808 15:59:58.988258 16226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:58.989140 16226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:58.990334 16226 hybrid_clock.cc:648] HybridClock initialized: now 1786204798990270 us; error 61 us; skew 500 ppm
I20260808 15:59:58.990643 16226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:58.992794 16226 webserver.cc:460] Webserver started at http://127.27.254.1:34387/ using document root <none> and password file <none>
I20260808 15:59:58.993523 16226 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:58.993633 16226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:58.993958 16226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:58.996615 16226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data/instance:
uuid: "7c080e806563466dacf37fab696e4b8d"
format_stamp: "Formatted at 2026-08-08 15:59:58 on dist-test-slave-srgv"
server_key: "654381832881e691a8cf738ec1a23b1f"
server_key_iv: "31d3b7e122129b7cbb4adcc859724502"
server_key_version: "encryptionkey@0"
I20260808 15:59:58.997387 16226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance:
uuid: "7c080e806563466dacf37fab696e4b8d"
format_stamp: "Formatted at 2026-08-08 15:59:58 on dist-test-slave-srgv"
server_key: "654381832881e691a8cf738ec1a23b1f"
server_key_iv: "31d3b7e122129b7cbb4adcc859724502"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.002641 16226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:59:59.005910 16240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.007133 16226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:59:59.007303 16226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal
uuid: "7c080e806563466dacf37fab696e4b8d"
format_stamp: "Formatted at 2026-08-08 15:59:58 on dist-test-slave-srgv"
server_key: "654381832881e691a8cf738ec1a23b1f"
server_key_iv: "31d3b7e122129b7cbb4adcc859724502"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.007520 16226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:59.029304 16226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:59.030115 16226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:59.030426 16226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:59.031419 16226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:59.032801 16226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:59.032929 16226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.033063 16226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:59.033151 16226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.065729 16226 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.1:33939
I20260808 15:59:59.065810 16354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.1:33939 every 8 connection(s)
I20260808 15:59:59.066991 16226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/data/info.pb
I20260808 15:59:59.074636 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16226
I20260808 15:59:59.074824 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal/instance
I20260808 15:59:59.075258 28664 external_mini_cluster.cc:1428] Setting key 4f69aba902abccbb82e559a4eb881135
I20260808 15:59:59.077952 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.2:0
--local_ip_for_outbound_sockets=127.27.254.2
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33315
--builtin_ntp_servers=127.27.254.20:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:59:59.079521 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:59.069495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.1:43117 (local address 127.27.254.62:33315)
0808 15:59:59.069991 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:59.069995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:59.070991 (+ 996us) server_negotiation.cc:407] Connection header received
0808 15:59:59.072847 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:59.072854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:59.072936 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:59.073057 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:59.074138 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.074945 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.076646 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.077191 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.077322 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:59.078719 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:59.078724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:59.078729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:59.078755 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:59.078794 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:59.078806 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:59.078899 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:59.079117 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 15:59:59.079207 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
I20260808 15:59:59.080492 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:59.069969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33315 (local address 127.27.254.1:43117)
0808 15:59:59.070814 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:59.070850 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:59.072566 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:59.073283 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:59.073290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:59.073379 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:59.073972 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:59.073989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.075113 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.075117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:59.076499 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:59.076511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.077441 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.077449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:59.077723 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:59.078426 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:59.078428 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:59.078464 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:59.078539 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:59.078897 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:59.078910 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:59.078919 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:59:59.079097 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 15:59:59.079455 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
I20260808 15:59:59.082129 16355 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33315
I20260808 15:59:59.082499 16355 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:59.083217 16355 heartbeater.cc:511] Master 127.27.254.62:33315 requested a full tablet report, sending...
I20260808 15:59:59.086159 16167 ts_manager.cc:194] Registered new tserver with Master: 7c080e806563466dacf37fab696e4b8d (127.27.254.1:33939)
I20260808 15:59:59.088327 16167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.1:43117
W20260808 15:59:59.404675 16359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:59.404924 16359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:59.405006 16359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:59.412187 16359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:59.412278 16359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:59.412380 16359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.2
I20260808 15:59:59.421455 16359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.254.2
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.2
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:59.422662 16359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:59.424305 16359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:59.437038 16364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:59.436954 16367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:59.437007 16365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:59.438084 16359 server_base.cc:1034] running on GCE node
I20260808 15:59:59.438638 16359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:59.439531 16359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:59.440770 16359 hybrid_clock.cc:648] HybridClock initialized: now 1786204799440731 us; error 49 us; skew 500 ppm
I20260808 15:59:59.441076 16359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:59.443217 16359 webserver.cc:460] Webserver started at http://127.27.254.2:35331/ using document root <none> and password file <none>
I20260808 15:59:59.443873 16359 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:59.444001 16359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:59.444319 16359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:59.447098 16359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data/instance:
uuid: "9ec4daf3eabe4f44825c0466d35022b8"
format_stamp: "Formatted at 2026-08-08 15:59:59 on dist-test-slave-srgv"
server_key: "02c87f0ba315d8b6a534b20672b852e6"
server_key_iv: "ac8c1d5ea6d9f30b8ccece5e04793592"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.447855 16359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance:
uuid: "9ec4daf3eabe4f44825c0466d35022b8"
format_stamp: "Formatted at 2026-08-08 15:59:59 on dist-test-slave-srgv"
server_key: "02c87f0ba315d8b6a534b20672b852e6"
server_key_iv: "ac8c1d5ea6d9f30b8ccece5e04793592"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.452672 16359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 15:59:59.455991 16373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.457239 16359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:59:59.457394 16359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal
uuid: "9ec4daf3eabe4f44825c0466d35022b8"
format_stamp: "Formatted at 2026-08-08 15:59:59 on dist-test-slave-srgv"
server_key: "02c87f0ba315d8b6a534b20672b852e6"
server_key_iv: "ac8c1d5ea6d9f30b8ccece5e04793592"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.457647 16359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:59.473500 16359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:59.474228 16359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:59.474509 16359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:59.475365 16359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:59.476675 16359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:59.476796 16359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.476928 16359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:59.477020 16359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.509691 16359 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.2:36825
I20260808 15:59:59.509749 16487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.2:36825 every 8 connection(s)
I20260808 15:59:59.511067 16359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/data/info.pb
I20260808 15:59:59.516283 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16359
I20260808 15:59:59.516469 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal/instance
I20260808 15:59:59.517007 28664 external_mini_cluster.cc:1428] Setting key 28e25521893ff29c8f1e982c589278cc
I20260808 15:59:59.519908 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.254.3:0
--local_ip_for_outbound_sockets=127.27.254.3
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33315
--builtin_ntp_servers=127.27.254.20:38195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:59:59.525611 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:59.513795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:53755 (local address 127.27.254.62:33315)
0808 15:59:59.514010 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:59.514016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:59.515445 (+ 1429us) server_negotiation.cc:407] Connection header received
0808 15:59:59.517957 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:59.517964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:59.518056 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:59.518178 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:59.519452 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.520232 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.522793 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.523198 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.523289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:59.524872 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:59.524876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:59.524879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:59.524896 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:59.524925 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:59.524936 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:59.525006 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:59.525247 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 15:59:59.525316 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:59:59.527055 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:59.514422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33315 (local address 127.27.254.2:53755)
0808 15:59:59.515296 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:59.515331 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:59.517566 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:59.518371 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:59.518379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:59.518461 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:59.519279 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:59.519305 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.520380 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.520389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:59.522647 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:59.522665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.523409 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.523418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:59.523758 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:59.524529 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:59.524534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:59.524585 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:59.524683 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:59.525032 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:59.525044 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:59.525054 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:59:59.525237 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 15:59:59.525684 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":283,"threads_started":1}
I20260808 15:59:59.528388 16488 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33315
I20260808 15:59:59.528680 16488 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:59.529485 16488 heartbeater.cc:511] Master 127.27.254.62:33315 requested a full tablet report, sending...
I20260808 15:59:59.530978 16167 ts_manager.cc:194] Registered new tserver with Master: 9ec4daf3eabe4f44825c0466d35022b8 (127.27.254.2:36825)
I20260808 15:59:59.531821 16167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.2:53755
W20260808 15:59:59.862949 16491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:59:59.863217 16491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:59:59.863301 16491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:59:59.870465 16491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:59:59.870595 16491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:59:59.870826 16491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.254.3
I20260808 15:59:59.880235 16491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.254.3
--webserver_port=0
--tserver_master_addrs=127.27.254.62:33315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.254.3
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:59:59.881626 16491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:59:59.883394 16491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:59:59.896238 16496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:59.896225 16499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:59:59.896291 16497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:59:59.897091 16491 server_base.cc:1034] running on GCE node
I20260808 15:59:59.897694 16491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:59:59.898523 16491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:59:59.899798 16491 hybrid_clock.cc:648] HybridClock initialized: now 1786204799899749 us; error 67 us; skew 500 ppm
I20260808 15:59:59.900133 16491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:59:59.902375 16491 webserver.cc:460] Webserver started at http://127.27.254.3:35325/ using document root <none> and password file <none>
I20260808 15:59:59.903055 16491 fs_manager.cc:359] Metadata directory not provided
I20260808 15:59:59.903185 16491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:59:59.903491 16491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:59:59.906255 16491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data/instance:
uuid: "0433e428980a4cc6bd6f6ca900f6d7e1"
format_stamp: "Formatted at 2026-08-08 15:59:59 on dist-test-slave-srgv"
server_key: "6f39a2df8bbc52d9c339cb30bd05e672"
server_key_iv: "4361c998215e1eafbb86e014793228dd"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.907020 16491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance:
uuid: "0433e428980a4cc6bd6f6ca900f6d7e1"
format_stamp: "Formatted at 2026-08-08 15:59:59 on dist-test-slave-srgv"
server_key: "6f39a2df8bbc52d9c339cb30bd05e672"
server_key_iv: "4361c998215e1eafbb86e014793228dd"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.911970 16491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260808 15:59:59.915517 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.916785 16491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:59:59.916952 16491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal
uuid: "0433e428980a4cc6bd6f6ca900f6d7e1"
format_stamp: "Formatted at 2026-08-08 15:59:59 on dist-test-slave-srgv"
server_key: "6f39a2df8bbc52d9c339cb30bd05e672"
server_key_iv: "4361c998215e1eafbb86e014793228dd"
server_key_version: "encryptionkey@0"
I20260808 15:59:59.917158 16491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:59:59.931322 16491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:59:59.932081 16491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:59:59.932379 16491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:59:59.933287 16491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:59:59.934724 16491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:59:59.934859 16491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.934979 16491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:59:59.935065 16491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:59:59.969283 16491 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.3:39355
I20260808 15:59:59.969336 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.3:39355 every 8 connection(s)
I20260808 15:59:59.970589 16491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/data/info.pb
I20260808 15:59:59.971086 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16491
I20260808 15:59:59.971288 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal/instance
I20260808 15:59:59.971895 28664 external_mini_cluster.cc:1428] Setting key 451388f5a19678f3e913e11a972fcc58
I20260808 15:59:59.982895 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:59.973180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.3:37643 (local address 127.27.254.62:33315)
0808 15:59:59.973396 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:59:59.973403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:59:59.974949 (+ 1546us) server_negotiation.cc:407] Connection header received
0808 15:59:59.976744 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:59:59.976748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:59:59.976809 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:59:59.976891 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:59:59.978015 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.978852 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.980276 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.980646 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.980741 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:59.981993 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:59:59.981998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:59:59.982001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:59:59.982017 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:59:59.982046 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:59:59.982057 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:59:59.982188 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:59:59.982474 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 15:59:59.982567 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:59:59.983773 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:59:59.973854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33315 (local address 127.27.254.3:37643)
0808 15:59:59.974767 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0808 15:59:59.974811 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:59:59.976504 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:59:59.977089 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:59:59.977097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:59:59.977180 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:59:59.977864 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:59.977881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.978973 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.978977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:59.980159 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:59:59.980170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:59:59.980818 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:59:59.980822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:59:59.981070 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:59:59.981731 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:59:59.981733 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:59:59.981771 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:59:59.981839 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:59:59.982186 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:59:59.982198 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:59:59.982208 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:59:59.982446 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 15:59:59.982794 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":304,"threads_started":1}
I20260808 15:59:59.985069 16619 heartbeater.cc:347] Connected to a master server at 127.27.254.62:33315
I20260808 15:59:59.985328 16619 heartbeater.cc:464] Registering TS with master...
I20260808 15:59:59.986021 16619 heartbeater.cc:511] Master 127.27.254.62:33315 requested a full tablet report, sending...
I20260808 15:59:59.987473 16166 ts_manager.cc:194] Registered new tserver with Master: 0433e428980a4cc6bd6f6ca900f6d7e1 (127.27.254.3:39355)
I20260808 15:59:59.988245 16166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.254.3:37643
I20260808 15:59:59.999328 28664 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 16:00:00.008703 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.003226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.27.254.62:33315)
0808 16:00:00.003490 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.003499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.003860 (+ 361us) server_negotiation.cc:407] Connection header received
0808 16:00:00.003987 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.003991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.004058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.004137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:00.004603 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.005275 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.006637 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.007029 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.007121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.008058 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:00.008063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:00.008066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:00.008083 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:00.008113 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:00.008124 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:00.008193 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.008309 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.008380 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 16:00:00.012616 16166 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 16:00:00.019654 16166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 16:00:00.056579 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.031960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33939 (local address 127.0.0.1:57236)
0808 16:00:00.046729 (+ 14769us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:00.046759 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:00.046901 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:00.048180 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:00.048188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:00.048222 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 16:00:00.048834 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.048850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.051374 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.051383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.053666 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.053678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.055612 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.055619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.055827 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.055883 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 16:00:00.056015 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 16:00:00.056202 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14621,"thread_start_us":12396,"threads_started":1}
I20260808 16:00:00.056605 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.032462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.27.254.1:33939)
0808 16:00:00.044550 (+ 12088us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.044562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.047208 (+ 2646us) server_negotiation.cc:407] Connection header received
0808 16:00:00.047336 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.047347 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.047636 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.047860 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 16:00:00.049083 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.051150 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.053866 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.055408 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.055564 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.055845 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.056076 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.056265 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11936,"thread_start_us":11825,"threads_started":1}
I20260808 16:00:00.057380 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.031925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.2:36825 (local address 127.0.0.1:41380)
0808 16:00:00.043826 (+ 11901us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:00.043892 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:00.044111 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:00.045853 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:00.045864 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:00.045898 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 16:00:00.046405 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.046425 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.050372 (+ 3947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.050380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.053863 (+ 3483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.053885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.056397 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.056403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.056610 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.056647 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 16:00:00.056931 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 16:00:00.057056 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11525,"mutex_wait_us":1,"thread_start_us":15045,"threads_started":1}
I20260808 16:00:00.057528 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.032357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.27.254.2:36825)
0808 16:00:00.044930 (+ 12573us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.044936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.044961 (+ 25us) server_negotiation.cc:407] Connection header received
0808 16:00:00.045044 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.045050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.045283 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.045545 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 16:00:00.046691 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.050134 (+ 3443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.054103 (+ 3969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.055887 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.056010 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.056225 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.056989 (+ 764us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.057143 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12420,"thread_start_us":12305,"threads_started":1}
I20260808 16:00:00.060694 16288 tablet_service.cc:1467] Processing CreateTablet for tablet aed94f9c0caf4f35aa462e79e259a881 (DEFAULT_TABLE table=test-table [id=b9771511efc245c39b9c2bd66f927a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 16:00:00.060860 16422 tablet_service.cc:1467] Processing CreateTablet for tablet aed94f9c0caf4f35aa462e79e259a881 (DEFAULT_TABLE table=test-table [id=b9771511efc245c39b9c2bd66f927a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 16:00:00.062111 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.045386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:39355 (local address 127.0.0.1:52318)
0808 16:00:00.052191 (+ 6805us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:00.052214 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:00.052355 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:00.055098 (+ 2743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:00.055105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:00.055129 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 16:00:00.055471 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.055487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.057436 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.057441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.059354 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.059369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.061175 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.061185 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.061356 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.061386 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 16:00:00.061489 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 16:00:00.061624 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6649,"thread_start_us":7028,"threads_started":1}
I20260808 16:00:00.062253 16288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed94f9c0caf4f35aa462e79e259a881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:00.062672 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.045756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.27.254.3:39355)
0808 16:00:00.054254 (+ 8498us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.054263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.054295 (+ 32us) server_negotiation.cc:407] Connection header received
0808 16:00:00.054393 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.054401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.054635 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.054838 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 16:00:00.055696 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.057231 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.059554 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.060987 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.061128 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.062032 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.062106 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.062295 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8305,"thread_start_us":8141,"threads_started":1}
I20260808 16:00:00.062839 16422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed94f9c0caf4f35aa462e79e259a881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:00.065732 16553 tablet_service.cc:1467] Processing CreateTablet for tablet aed94f9c0caf4f35aa462e79e259a881 (DEFAULT_TABLE table=test-table [id=b9771511efc245c39b9c2bd66f927a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 16:00:00.067092 16553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed94f9c0caf4f35aa462e79e259a881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:00.077531 16637 tablet_bootstrap.cc:492] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Bootstrap starting.
I20260808 16:00:00.079782 16638 tablet_bootstrap.cc:492] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Bootstrap starting.
I20260808 16:00:00.081359 16637 tablet_bootstrap.cc:654] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:00.082919 16637 log.cc:824] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:00.083196 16638 tablet_bootstrap.cc:654] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:00.084417 16638 log.cc:824] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:00.084446 16639 tablet_bootstrap.cc:492] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Bootstrap starting.
I20260808 16:00:00.086166 16637 tablet_bootstrap.cc:492] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: No bootstrap required, opened a new log
I20260808 16:00:00.086483 16637 ts_tablet_manager.cc:1397] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 16:00:00.087819 16638 tablet_bootstrap.cc:492] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: No bootstrap required, opened a new log
I20260808 16:00:00.087939 16639 tablet_bootstrap.cc:654] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:00.088081 16638 ts_tablet_manager.cc:1397] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260808 16:00:00.088922 16639 log.cc:824] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:00.091964 16639 tablet_bootstrap.cc:492] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: No bootstrap required, opened a new log
I20260808 16:00:00.091964 16355 heartbeater.cc:503] Master 127.27.254.62:33315 was elected leader, sending a full tablet report...
I20260808 16:00:00.092321 16639 ts_tablet_manager.cc:1397] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 16:00:00.099041 16637 raft_consensus.cc:348] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.099042 16638 raft_consensus.cc:348] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.099522 16638 raft_consensus.cc:374] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:00.099517 16637 raft_consensus.cc:374] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:00.099617 16637 raft_consensus.cc:729] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec4daf3eabe4f44825c0466d35022b8, State: Initialized, Role: FOLLOWER
I20260808 16:00:00.099648 16638 raft_consensus.cc:729] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c080e806563466dacf37fab696e4b8d, State: Initialized, Role: FOLLOWER
I20260808 16:00:00.100303 16637 consensus_queue.cc:260] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.100303 16638 consensus_queue.cc:260] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.101783 16639 raft_consensus.cc:348] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.102274 16639 raft_consensus.cc:374] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:00.102358 16637 ts_tablet_manager.cc:1428] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260808 16:00:00.102396 16639 raft_consensus.cc:729] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0433e428980a4cc6bd6f6ca900f6d7e1, State: Initialized, Role: FOLLOWER
I20260808 16:00:00.102541 16638 ts_tablet_manager.cc:1428] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260808 16:00:00.103070 16488 heartbeater.cc:503] Master 127.27.254.62:33315 was elected leader, sending a full tablet report...
I20260808 16:00:00.103044 16639 consensus_queue.cc:260] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.104110 16619 heartbeater.cc:503] Master 127.27.254.62:33315 was elected leader, sending a full tablet report...
I20260808 16:00:00.104773 16639 ts_tablet_manager.cc:1428] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260808 16:00:00.222378 16620 tablet.cc:2369] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 16:00:00.263152 16489 tablet.cc:2369] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 16:00:00.319481 16356 tablet.cc:2369] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 16:00:00.338557 16643 raft_consensus.cc:482] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 16:00:00.338873 16643 raft_consensus.cc:504] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.340448 16643 leader_election.cc:288] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c080e806563466dacf37fab696e4b8d (127.27.254.1:33939), 0433e428980a4cc6bd6f6ca900f6d7e1 (127.27.254.3:39355)
I20260808 16:00:00.347535 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.341010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:47455 (local address 127.27.254.1:33939)
0808 16:00:00.341253 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.341261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.341566 (+ 305us) server_negotiation.cc:407] Connection header received
0808 16:00:00.341843 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.341850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.341956 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.342059 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 16:00:00.342714 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.343613 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.345666 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.346513 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.346636 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.346684 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.347132 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.347273 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 16:00:00.347553 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.341736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.254.2:51385 (local address 127.27.254.3:39355)
0808 16:00:00.341923 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.341927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.342359 (+ 432us) server_negotiation.cc:407] Connection header received
0808 16:00:00.342473 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.342481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.342563 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.342664 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 16:00:00.343216 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.344059 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.345909 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.346704 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.346859 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.346926 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.347197 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.347337 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 16:00:00.347573 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.340783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.1:33939 (local address 127.27.254.2:47455)
0808 16:00:00.341460 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:00.341481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:00.341680 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:00.342119 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:00.342130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:00.342156 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 16:00:00.342552 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.342565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.343747 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.343760 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.345477 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.345489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.346703 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.346710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.346927 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.346978 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 16:00:00.347053 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 16:00:00.347179 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":103,"thread_start_us":319,"threads_started":1}
I20260808 16:00:00.347633 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.341528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.3:39355 (local address 127.27.254.2:51385)
0808 16:00:00.342255 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:00.342269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:00.342366 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:00.342696 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:00.342701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:00.342723 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 16:00:00.343069 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.343080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.344193 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.344196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.345712 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:00.345727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.346884 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.346888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:00.347009 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.347055 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 16:00:00.347147 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 16:00:00.347262 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":48,"thread_start_us":374,"threads_started":1}
I20260808 16:00:00.348637 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed94f9c0caf4f35aa462e79e259a881" candidate_uuid: "9ec4daf3eabe4f44825c0466d35022b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" is_pre_election: true
I20260808 16:00:00.348670 16309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed94f9c0caf4f35aa462e79e259a881" candidate_uuid: "9ec4daf3eabe4f44825c0466d35022b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c080e806563466dacf37fab696e4b8d" is_pre_election: true
I20260808 16:00:00.349139 16309 raft_consensus.cc:2454] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ec4daf3eabe4f44825c0466d35022b8 in term 0.
I20260808 16:00:00.349139 16573 raft_consensus.cc:2454] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ec4daf3eabe4f44825c0466d35022b8 in term 0.
I20260808 16:00:00.349853 16374 leader_election.cc:302] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0433e428980a4cc6bd6f6ca900f6d7e1, 9ec4daf3eabe4f44825c0466d35022b8; no voters:
I20260808 16:00:00.350216 16643 raft_consensus.cc:2781] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 16:00:00.350353 16643 raft_consensus.cc:482] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 16:00:00.350497 16643 raft_consensus.cc:3037] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:00.352628 16643 raft_consensus.cc:504] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.353173 16643 leader_election.cc:288] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [CANDIDATE]: Term 1 election: Requested vote from peers 7c080e806563466dacf37fab696e4b8d (127.27.254.1:33939), 0433e428980a4cc6bd6f6ca900f6d7e1 (127.27.254.3:39355)
I20260808 16:00:00.353683 16309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed94f9c0caf4f35aa462e79e259a881" candidate_uuid: "9ec4daf3eabe4f44825c0466d35022b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c080e806563466dacf37fab696e4b8d"
I20260808 16:00:00.353763 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aed94f9c0caf4f35aa462e79e259a881" candidate_uuid: "9ec4daf3eabe4f44825c0466d35022b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1"
I20260808 16:00:00.353878 16309 raft_consensus.cc:3037] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:00.353989 16573 raft_consensus.cc:3037] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:00.356058 16309 raft_consensus.cc:2454] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec4daf3eabe4f44825c0466d35022b8 in term 1.
I20260808 16:00:00.356061 16573 raft_consensus.cc:2454] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec4daf3eabe4f44825c0466d35022b8 in term 1.
I20260808 16:00:00.356437 16374 leader_election.cc:302] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0433e428980a4cc6bd6f6ca900f6d7e1, 9ec4daf3eabe4f44825c0466d35022b8; no voters:
I20260808 16:00:00.356758 16643 raft_consensus.cc:2781] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:00.357146 16643 raft_consensus.cc:686] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec4daf3eabe4f44825c0466d35022b8, State: Running, Role: LEADER
I20260808 16:00:00.357662 16643 consensus_queue.cc:237] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } }
I20260808 16:00:00.362304 16166 catalog_manager.cc:5495] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ec4daf3eabe4f44825c0466d35022b8 (127.27.254.2). New cstate: current_term: 1 leader_uuid: "9ec4daf3eabe4f44825c0466d35022b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec4daf3eabe4f44825c0466d35022b8" member_type: VOTER last_known_addr { host: "127.27.254.2" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 } health_report { overall_health: UNKNOWN } } }
I20260808 16:00:00.410228 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.404826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.27.254.2:36825)
0808 16:00:00.405055 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.405061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.405078 (+ 17us) server_negotiation.cc:407] Connection header received
0808 16:00:00.405168 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.405172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.405272 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.405449 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:00.405828 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.406549 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.408036 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.408440 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.408567 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.409076 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:00.409640 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:00.409796 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.409891 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.409969 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 16:00:00.418256 16309 raft_consensus.cc:1261] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 1 FOLLOWER]: Refusing update from remote peer 9ec4daf3eabe4f44825c0466d35022b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 16:00:00.418262 16573 raft_consensus.cc:1261] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 1 FOLLOWER]: Refusing update from remote peer 9ec4daf3eabe4f44825c0466d35022b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 16:00:00.419122 16648 consensus_queue.cc:1035] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c080e806563466dacf37fab696e4b8d" member_type: VOTER last_known_addr { host: "127.27.254.1" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 16:00:00.419526 16643 consensus_queue.cc:1035] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0433e428980a4cc6bd6f6ca900f6d7e1" member_type: VOTER last_known_addr { host: "127.27.254.3" port: 39355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 16:00:00.433746 16652 mvcc.cc:205] Tried to move back new op lower bound from 7316294862503276544 to 7316294862268973056. Current Snapshot: MvccSnapshot[applied={T|T < 7316294862503276544}]
I20260808 16:00:00.443645 16166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 16:00:00.443933 16166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 16:00:00.447564 16166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 976f73db29074b61a498ae9528e5b00d: Sending DeleteTablet for 3 replicas of tablet aed94f9c0caf4f35aa462e79e259a881
I20260808 16:00:00.448848 16288 tablet_service.cc:1514] Processing DeleteTablet for tablet aed94f9c0caf4f35aa462e79e259a881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 16:00:00 UTC) from {username='slave'} at 127.0.0.1:57236
I20260808 16:00:00.448891 16553 tablet_service.cc:1514] Processing DeleteTablet for tablet aed94f9c0caf4f35aa462e79e259a881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 16:00:00 UTC) from {username='slave'} at 127.0.0.1:52318
I20260808 16:00:00.449821 16422 tablet_service.cc:1514] Processing DeleteTablet for tablet aed94f9c0caf4f35aa462e79e259a881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 16:00:00 UTC) from {username='slave'} at 127.0.0.1:41380
I20260808 16:00:00.449882 16662 tablet_replica.cc:330] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: stopping tablet replica
I20260808 16:00:00.450399 16662 raft_consensus.cc:2229] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 16:00:00.450989 16662 raft_consensus.cc:2258] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 16:00:00.451826 16661 tablet_replica.cc:330] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: stopping tablet replica
I20260808 16:00:00.452000 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16226
I20260808 16:00:00.452067 16663 tablet_replica.cc:330] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: stopping tablet replica
I20260808 16:00:00.452466 16663 raft_consensus.cc:2229] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 16:00:00.453172 16663 raft_consensus.cc:2258] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 16:00:00.453770 16662 ts_tablet_manager.cc:1905] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 16:00:00.455284 16663 ts_tablet_manager.cc:1905] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 16:00:00.460723 16662 ts_tablet_manager.cc:1918] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 16:00:00.461264 16661 raft_consensus.cc:2229] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 16:00:00.461841 16661 raft_consensus.cc:2258] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 16:00:00.463304 16663 ts_tablet_manager.cc:1918] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 16:00:00.463548 16663 log.cc:1196] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-1/wal/wals/aed94f9c0caf4f35aa462e79e259a881
I20260808 16:00:00.464030 16661 ts_tablet_manager.cc:1905] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 16:00:00.464309 16663 ts_tablet_manager.cc:1939] T aed94f9c0caf4f35aa462e79e259a881 P 9ec4daf3eabe4f44825c0466d35022b8: Deleting consensus metadata
I20260808 16:00:00.466198 16154 catalog_manager.cc:4841] TS 9ec4daf3eabe4f44825c0466d35022b8 (127.27.254.2:36825): tablet aed94f9c0caf4f35aa462e79e259a881 (table test-table [id=b9771511efc245c39b9c2bd66f927a9b]) successfully deleted
I20260808 16:00:00.467031 16662 log.cc:1196] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-0/wal/wals/aed94f9c0caf4f35aa462e79e259a881
I20260808 16:00:00.467705 16662 ts_tablet_manager.cc:1939] T aed94f9c0caf4f35aa462e79e259a881 P 7c080e806563466dacf37fab696e4b8d: Deleting consensus metadata
I20260808 16:00:00.469758 16151 catalog_manager.cc:4841] TS 7c080e806563466dacf37fab696e4b8d (127.27.254.1:33939): tablet aed94f9c0caf4f35aa462e79e259a881 (table test-table [id=b9771511efc245c39b9c2bd66f927a9b]) successfully deleted
I20260808 16:00:00.471002 16661 ts_tablet_manager.cc:1918] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 16:00:00.471155 16661 log.cc:1196] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Deleting WAL directory at /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204485174398-28664-0/minicluster-data/ts-2/wal/wals/aed94f9c0caf4f35aa462e79e259a881
I20260808 16:00:00.471648 16661 ts_tablet_manager.cc:1939] T aed94f9c0caf4f35aa462e79e259a881 P 0433e428980a4cc6bd6f6ca900f6d7e1: Deleting consensus metadata
I20260808 16:00:00.473286 16151 catalog_manager.cc:4841] TS 0433e428980a4cc6bd6f6ca900f6d7e1 (127.27.254.3:39355): tablet aed94f9c0caf4f35aa462e79e259a881 (table test-table [id=b9771511efc245c39b9c2bd66f927a9b]) successfully deleted
I20260808 16:00:00.479621 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.453374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.27.254.1:33939)
0808 16:00:00.453560 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.453568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.462677 (+ 9109us) server_negotiation.cc:407] Connection header received
0808 16:00:00.463009 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.463017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.463140 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.463225 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:00.463973 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.465098 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.466911 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.467632 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.467787 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.478045 (+ 10258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:00.478053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:00.478066 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:00.478675 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:00.478723 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:00.478745 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:00.478874 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.479021 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.479182 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 16:00:00.480495 16350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:00.594841 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16226
I20260808 16:00:00.613227 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16359
I20260808 16:00:00.621671 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.614145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.27.254.2:36825)
0808 16:00:00.614342 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.614347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.614423 (+ 76us) server_negotiation.cc:407] Connection header received
0808 16:00:00.614681 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.614685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.614795 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.614887 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:00.615349 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.616112 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.618102 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.618520 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.618613 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.619878 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:00.619887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:00.619901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:00.620738 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:00.620787 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:00.620820 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:00.620967 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.621098 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.621183 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 16:00:00.622450 16483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:00.734894 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16359
I20260808 16:00:00.754734 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16491
I20260808 16:00:00.761569 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:00.755471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.27.254.3:39355)
0808 16:00:00.755660 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:00.755664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:00.755683 (+ 19us) server_negotiation.cc:407] Connection header received
0808 16:00:00.755834 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:00.755838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:00.755919 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:00.756050 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:00.756382 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.757222 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.758578 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:00.759089 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:00.759183 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:00.760194 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:00.760203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:00.760217 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:00.760805 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:00.760852 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:00.760870 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:00.760960 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:00.761057 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 16:00:00.761140 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 16:00:00.762212 16614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:00.850548 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16491
I20260808 16:00:00.870893 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16136
I20260808 16:00:00.872007 16197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:00.980675 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16136
2026-08-08T16:00:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2922 ms)
[----------] 6 tests from EncryptionPolicyTest (21376 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:01 dist-test-slave-srgv krb5kdc[16671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:01 dist-test-slave-srgv krb5kdc[16671](info): set up 2 sockets
Aug 08 16:00:01 dist-test-slave-srgv krb5kdc[16671](info): commencing operation
krb5kdc: starting...
W20260808 16:00:02.083863 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.010s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:02 dist-test-slave-srgv krb5kdc[16671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:02.325629 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:02.645287 16684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:02.645634 16684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:02.645740 16684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:02.652606 16684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:02.652709 16684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:02.652755 16684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:02.652813 16684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:02.652856 16684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:02.662294 16684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:02.663441 16684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:02.664656 16684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:02.676591 16689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:02.676642 16692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:02.676633 16690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:02.677541 16684 server_base.cc:1034] running on GCE node
I20260808 16:00:02.678257 16684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:02.679572 16684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:02.680799 16684 hybrid_clock.cc:648] HybridClock initialized: now 1786204802680767 us; error 44 us; skew 500 ppm
I20260808 16:00:02.681138 16684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:02 dist-test-slave-srgv krb5kdc[16671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:02.684154 16684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:02.685588 16684 webserver.cc:460] Webserver started at http://127.0.0.1:40611/ using document root <none> and password file <none>
I20260808 16:00:02.686174 16684 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:02.686280 16684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:02.686517 16684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:02.689276 16684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "16427e9ff0da4e0a9d96ac3668c61e0d"
format_stamp: "Formatted at 2026-08-08 16:00:02 on dist-test-slave-srgv"
server_key: "36772f028e51c26c70b36fc147312b3f"
server_key_iv: "452af4fbd92624265bd0143345d298ad"
server_key_version: "encryptionkey@0"
I20260808 16:00:02.690044 16684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "16427e9ff0da4e0a9d96ac3668c61e0d"
format_stamp: "Formatted at 2026-08-08 16:00:02 on dist-test-slave-srgv"
server_key: "36772f028e51c26c70b36fc147312b3f"
server_key_iv: "452af4fbd92624265bd0143345d298ad"
server_key_version: "encryptionkey@0"
I20260808 16:00:02.694782 16684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 16:00:02.698004 16699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:02.699239 16684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 16:00:02.699424 16684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "16427e9ff0da4e0a9d96ac3668c61e0d"
format_stamp: "Formatted at 2026-08-08 16:00:02 on dist-test-slave-srgv"
server_key: "36772f028e51c26c70b36fc147312b3f"
server_key_iv: "452af4fbd92624265bd0143345d298ad"
server_key_version: "encryptionkey@0"
I20260808 16:00:02.699599 16684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:02.719378 16684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:02.723170 16684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:02.723448 16684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:02.741291 16684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33861
I20260808 16:00:02.741302 16750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33861 every 8 connection(s)
I20260808 16:00:02.742596 16684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:02.746111 16751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:02.748232 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16684
I20260808 16:00:02.748457 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:02.749081 28664 external_mini_cluster.cc:1428] Setting key 1c5d0528a47be8465a9945eb6d1b0115
I20260808 16:00:02.758613 16751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d: Bootstrap starting.
I20260808 16:00:02.762055 16751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:02.762992 16751 log.cc:824] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d: Log is configured to *not* fsync() on all Append() calls
Aug 08 16:00:02 dist-test-slave-srgv krb5kdc[16671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 16:00:02.765604 16751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d: No bootstrap required, opened a new log
I20260808 16:00:02.772163 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:02.751125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39037 (local address 127.0.0.1:33861)
0808 16:00:02.751912 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:02.751927 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:02.751972 (+ 45us) server_negotiation.cc:407] Connection header received
0808 16:00:02.754267 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:02.754296 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:02.754799 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:02.755227 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:02.756510 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:02.758441 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:02.759674 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:02.760757 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:02.760924 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:02.764999 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:02.765007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:02.765020 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:02.765051 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:02.768116 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:02.768915 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:02.768921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:02.768926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:02.769081 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:02.769618 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:02.769622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:02.769624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:02.770122 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:02.770452 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:02.770949 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 16:00:02.771236 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":310,"threads_started":1}
I20260808 16:00:02.773236 16751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260808 16:00:02.773710 16751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:02.773821 16751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16427e9ff0da4e0a9d96ac3668c61e0d, State: Initialized, Role: FOLLOWER
I20260808 16:00:02.774423 16751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260808 16:00:02.774740 16751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:02.774900 16751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:02.775071 16751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:02.776147 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:02.777565 16751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260808 16:00:02.778024 16751 leader_election.cc:302] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16427e9ff0da4e0a9d96ac3668c61e0d; no voters:
I20260808 16:00:02.778581 16751 leader_election.cc:288] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:02.778729 16757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:02.780416 16751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:02.785786 16757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [term 1 LEADER]: Becoming Leader. State: Replica: 16427e9ff0da4e0a9d96ac3668c61e0d, State: Running, Role: LEADER
I20260808 16:00:02.786463 16757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260808 16:00:02.789714 16759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } } }
I20260808 16:00:02.790160 16759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:02.791509 16760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16427e9ff0da4e0a9d96ac3668c61e0d. Latest consensus state: current_term: 1 leader_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16427e9ff0da4e0a9d96ac3668c61e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } } }
I20260808 16:00:02.791872 16760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:02.795092 16764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:02.800606 16764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:02.811782 16764 catalog_manager.cc:1347] Generated new cluster ID: dbf418a84f8c4a24b3e9eaf4cb79f9de
I20260808 16:00:02.811877 16764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:02.813854 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:02.794845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55935 (local address 127.0.0.1:33861)
0808 16:00:02.795093 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:02.795102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:02.796055 (+ 953us) server_negotiation.cc:407] Connection header received
0808 16:00:02.796234 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:02.796240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:02.796313 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:02.796431 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:02.798627 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:02.799817 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:02.801711 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:02.802392 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:02.802510 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:02.805908 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:02.805915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:02.805919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:02.805955 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:02.809723 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:02.810665 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:02.810670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:02.810673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:02.810844 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:02.811408 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:02.811413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:02.811415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:02.811904 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:02.812172 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:02.813331 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0808 16:00:02.813520 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 16:00:02.836215 16764 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:02.837954 16764 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:02.847944 16764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16427e9ff0da4e0a9d96ac3668c61e0d: Generated new TSK 0
I20260808 16:00:02.848683 16764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:02.861877 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16684
I20260808 16:00:02.863925 16746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:02.953128 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16684
W20260808 16:00:02.970559 16669 connection.cc:535] client connection to 127.0.0.1:33861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33861): Connection reset by peer (error 104)
2026-08-08T16:00:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:03 dist-test-slave-srgv krb5kdc[16784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:03 dist-test-slave-srgv krb5kdc[16784](info): set up 2 sockets
Aug 08 16:00:03 dist-test-slave-srgv krb5kdc[16784](info): commencing operation
krb5kdc: starting...
W20260808 16:00:04.063549 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:04 dist-test-slave-srgv krb5kdc[16784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:04.321282 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:04.638482 16797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:04.639048 16797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:04.639236 16797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:04.646232 16797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:04.646384 16797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:04.646485 16797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:04.646593 16797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:04.646674 16797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:04.656116 16797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:04.657351 16797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:04.658975 16797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:04.670449 16802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:04.670357 16803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:04.670394 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:04.671473 16797 server_base.cc:1034] running on GCE node
I20260808 16:00:04.672165 16797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:04.673517 16797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:04.674741 16797 hybrid_clock.cc:648] HybridClock initialized: now 1786204804674697 us; error 51 us; skew 500 ppm
I20260808 16:00:04.675077 16797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:04 dist-test-slave-srgv krb5kdc[16784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:04.678154 16797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:04.679522 16797 webserver.cc:460] Webserver started at http://127.0.0.1:42009/ using document root <none> and password file <none>
I20260808 16:00:04.680192 16797 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:04.680344 16797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:04.680644 16797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:04.683548 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "bb0dc396b471442ba4df050e0c4c57f6"
format_stamp: "Formatted at 2026-08-08 16:00:04 on dist-test-slave-srgv"
server_key: "f1b16215db26458d044ea638b40d02b8"
server_key_iv: "384ae054280d6af631c8f38566273281"
server_key_version: "encryptionkey@0"
I20260808 16:00:04.684310 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "bb0dc396b471442ba4df050e0c4c57f6"
format_stamp: "Formatted at 2026-08-08 16:00:04 on dist-test-slave-srgv"
server_key: "f1b16215db26458d044ea638b40d02b8"
server_key_iv: "384ae054280d6af631c8f38566273281"
server_key_version: "encryptionkey@0"
I20260808 16:00:04.689240 16797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 16:00:04.692616 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:04.694082 16797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:04.694263 16797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "bb0dc396b471442ba4df050e0c4c57f6"
format_stamp: "Formatted at 2026-08-08 16:00:04 on dist-test-slave-srgv"
server_key: "f1b16215db26458d044ea638b40d02b8"
server_key_iv: "384ae054280d6af631c8f38566273281"
server_key_version: "encryptionkey@0"
I20260808 16:00:04.694460 16797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:04.736007 16797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:04.740540 16797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:04.740830 16797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:04.758510 16797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42393
I20260808 16:00:04.758553 16863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42393 every 8 connection(s)
I20260808 16:00:04.759797 16797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:04.763329 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:04.766121 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16797
I20260808 16:00:04.766319 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:04.766840 28664 external_mini_cluster.cc:1428] Setting key db9b483ff10c6fa72e648c129e272892
I20260808 16:00:04.774801 16864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6: Bootstrap starting.
I20260808 16:00:04.778215 16864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:04.779289 16864 log.cc:824] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:04.779654 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:04.768559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46545 (local address 127.0.0.1:42393)
0808 16:00:04.769284 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:04.769294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:04.769331 (+ 37us) server_negotiation.cc:407] Connection header received
0808 16:00:04.771042 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:04.771061 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:04.771419 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:04.771709 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:04.772248 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:04.773631 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:04.775450 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:04.776049 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:04.776184 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:04.777548 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:04.777556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:04.777568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:04.778033 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:04.778098 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:04.778122 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:04.778245 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:04.778530 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 16:00:04.778736 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":343,"threads_started":1}
I20260808 16:00:04.781956 16864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6: No bootstrap required, opened a new log
I20260808 16:00:04.783337 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:04.792469 16864 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0dc396b471442ba4df050e0c4c57f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260808 16:00:04.792937 16864 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:04.793033 16864 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0dc396b471442ba4df050e0c4c57f6, State: Initialized, Role: FOLLOWER
I20260808 16:00:04.793694 16864 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0dc396b471442ba4df050e0c4c57f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260808 16:00:04.793946 16864 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:04.794107 16864 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:04.794296 16864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:04.796345 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:04.788564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59047 (local address 127.0.0.1:42393)
0808 16:00:04.788792 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:04.788799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:04.789328 (+ 529us) server_negotiation.cc:407] Connection header received
0808 16:00:04.789666 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:04.789672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:04.789758 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:04.789864 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:04.790542 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:04.791713 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:04.793602 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:04.794164 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:04.794330 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:04.795591 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:04.795597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:04.795603 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:04.795629 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:04.795699 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:04.795714 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:04.795838 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:04.796029 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 16:00:04.796079 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 16:00:04.796725 16864 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0dc396b471442ba4df050e0c4c57f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260808 16:00:04.797292 16864 leader_election.cc:302] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0dc396b471442ba4df050e0c4c57f6; no voters:
I20260808 16:00:04.797951 16864 leader_election.cc:288] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:04.798061 16874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:04.798803 16874 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [term 1 LEADER]: Becoming Leader. State: Replica: bb0dc396b471442ba4df050e0c4c57f6, State: Running, Role: LEADER
I20260808 16:00:04.799356 16874 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0dc396b471442ba4df050e0c4c57f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260808 16:00:04.799938 16864 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:04.807291 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0dc396b471442ba4df050e0c4c57f6. Latest consensus state: current_term: 1 leader_uuid: "bb0dc396b471442ba4df050e0c4c57f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0dc396b471442ba4df050e0c4c57f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260808 16:00:04.807274 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0dc396b471442ba4df050e0c4c57f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0dc396b471442ba4df050e0c4c57f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260808 16:00:04.807824 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:04.807857 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6 [sys.catalog]: This master's current role is: LEADER
W20260808 16:00:04.811408 16888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 16:00:04.811527 16888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 16:00:04.811626 16889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:04.813647 16889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:04.821511 16889 catalog_manager.cc:1347] Generated new cluster ID: 770c723c6de34509b9e190258d7975ff
I20260808 16:00:04.821607 16889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:04.843829 16889 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:04.845041 16889 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:04.852505 16889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0dc396b471442ba4df050e0c4c57f6: Generated new TSK 0
I20260808 16:00:04.853283 16889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:04.867970 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16797
W20260808 16:00:04.869132 16859 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46545
W20260808 16:00:04.869951 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16797: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:04.870129 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16797
W20260808 16:00:04.886538 16782 connection.cc:535] client connection to 127.0.0.1:42393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42393): Connection reset by peer (error 104)
2026-08-08T16:00:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:04 dist-test-slave-srgv krb5kdc[16896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:04 dist-test-slave-srgv krb5kdc[16896](info): set up 2 sockets
Aug 08 16:00:04 dist-test-slave-srgv krb5kdc[16896](info): commencing operation
krb5kdc: starting...
W20260808 16:00:05.975775 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:06 dist-test-slave-srgv krb5kdc[16896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:06.215919 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:06.526981 16909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:06.527248 16909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:06.527424 16909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:06.534644 16909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:06.534775 16909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:06.534878 16909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:06.534976 16909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:06.535054 16909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:06.544700 16909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:06.546048 16909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:06.547313 16909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:06.558945 16914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:06.559059 16915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:06.559088 16917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:06.560120 16909 server_base.cc:1034] running on GCE node
I20260808 16:00:06.560876 16909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:06.562250 16909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:06.563449 16909 hybrid_clock.cc:648] HybridClock initialized: now 1786204806563397 us; error 52 us; skew 500 ppm
I20260808 16:00:06.563787 16909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:06 dist-test-slave-srgv krb5kdc[16896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:06.571059 16909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:06.572415 16909 webserver.cc:460] Webserver started at http://127.0.0.1:44431/ using document root <none> and password file <none>
I20260808 16:00:06.573035 16909 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:06.573132 16909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:06.573386 16909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:06.576253 16909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "2945e0e27fd2455ea4def5326b703651"
format_stamp: "Formatted at 2026-08-08 16:00:06 on dist-test-slave-srgv"
server_key: "c0780f07b4a7e883e71763a690ea7114"
server_key_iv: "b36b64b8db9c9868cadd9a1ae77913d6"
server_key_version: "encryptionkey@0"
I20260808 16:00:06.576957 16909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "2945e0e27fd2455ea4def5326b703651"
format_stamp: "Formatted at 2026-08-08 16:00:06 on dist-test-slave-srgv"
server_key: "c0780f07b4a7e883e71763a690ea7114"
server_key_iv: "b36b64b8db9c9868cadd9a1ae77913d6"
server_key_version: "encryptionkey@0"
I20260808 16:00:06.581907 16909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 16:00:06.585033 16924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:06.586292 16909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:06.586450 16909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "2945e0e27fd2455ea4def5326b703651"
format_stamp: "Formatted at 2026-08-08 16:00:06 on dist-test-slave-srgv"
server_key: "c0780f07b4a7e883e71763a690ea7114"
server_key_iv: "b36b64b8db9c9868cadd9a1ae77913d6"
server_key_version: "encryptionkey@0"
I20260808 16:00:06.586635 16909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:06.591763 16909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:06.595820 16909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:06.596127 16909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:06.614356 16909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43475
I20260808 16:00:06.614387 16975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43475 every 8 connection(s)
I20260808 16:00:06.615681 16909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:06.617511 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 16909
I20260808 16:00:06.617717 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:06.618278 28664 external_mini_cluster.cc:1428] Setting key ea52252d9e8dc2a9cd3d498cbac05b3e
I20260808 16:00:06.620060 16976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:06.626127 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:06.620008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56951 (local address 127.0.0.1:43475)
0808 16:00:06.620856 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:06.620867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:06.620906 (+ 39us) server_negotiation.cc:407] Connection header received
0808 16:00:06.622531 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:06.622551 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:06.623109 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:06.623452 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:06.624544 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:06.624550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:06.624562 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:06.624575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:06.624653 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:06.624671 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:06.624843 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:06.624965 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 16:00:06.625175 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":393,"threads_started":1}
I20260808 16:00:06.629542 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:06.633270 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651: Bootstrap starting.
I20260808 16:00:06.637319 16976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:06.637362 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:06.634936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49945 (local address 127.0.0.1:43475)
0808 16:00:06.635083 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:06.635088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:06.635510 (+ 422us) server_negotiation.cc:407] Connection header received
0808 16:00:06.635672 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:06.635675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:06.635726 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:06.635800 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:06.636731 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:06.636735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:06.636738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:06.636751 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:06.636786 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:06.636800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:06.636884 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:06.637037 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 16:00:06.637077 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 16:00:06.638424 16976 log.cc:824] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:06.640883 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651: No bootstrap required, opened a new log
I20260808 16:00:06.647979 16976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2945e0e27fd2455ea4def5326b703651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260808 16:00:06.648378 16976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:06.648499 16976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2945e0e27fd2455ea4def5326b703651, State: Initialized, Role: FOLLOWER
I20260808 16:00:06.649106 16976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2945e0e27fd2455ea4def5326b703651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260808 16:00:06.649349 16976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:06.649536 16976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:06.649732 16976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:06.651759 16976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2945e0e27fd2455ea4def5326b703651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260808 16:00:06.652231 16976 leader_election.cc:302] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2945e0e27fd2455ea4def5326b703651; no voters:
I20260808 16:00:06.652802 16976 leader_election.cc:288] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:06.652937 16986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:06.653656 16986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [term 1 LEADER]: Becoming Leader. State: Replica: 2945e0e27fd2455ea4def5326b703651, State: Running, Role: LEADER
I20260808 16:00:06.654211 16986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2945e0e27fd2455ea4def5326b703651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260808 16:00:06.654893 16976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:06.657145 16988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2945e0e27fd2455ea4def5326b703651. Latest consensus state: current_term: 1 leader_uuid: "2945e0e27fd2455ea4def5326b703651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2945e0e27fd2455ea4def5326b703651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } } }
I20260808 16:00:06.657565 16988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:06.661082 16987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2945e0e27fd2455ea4def5326b703651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2945e0e27fd2455ea4def5326b703651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } } }
I20260808 16:00:06.663050 16987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:06.664067 16993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:06.669323 16993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:06.677053 16993 catalog_manager.cc:1347] Generated new cluster ID: 2e596823a4ce41d48f3ae8c2945e84d0
I20260808 16:00:06.677157 16993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:06.694353 16993 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:06.694886 16993 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:06.705111 16993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2945e0e27fd2455ea4def5326b703651: Generated new TSK 0
I20260808 16:00:06.706013 16993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:06.710309 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16909
W20260808 16:00:06.711301 16971 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56951
W20260808 16:00:06.712023 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 16909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:06.712186 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 16909
2026-08-08T16:00:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:06 dist-test-slave-srgv krb5kdc[17007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:06 dist-test-slave-srgv krb5kdc[17007](info): set up 2 sockets
Aug 08 16:00:06 dist-test-slave-srgv krb5kdc[17007](info): commencing operation
krb5kdc: starting...
W20260808 16:00:07.807881 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:07 dist-test-slave-srgv krb5kdc[17007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:08.057471 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:08.375838 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:08.376250 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:08.376428 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:08.383615 17020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:08.383719 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:08.383838 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:08.383937 17020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:08.384019 17020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:08.393352 17020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:08.394582 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:08.395824 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:08.406611 17026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:08.406813 17025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:08.407449 17028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:08.408439 17020 server_base.cc:1034] running on GCE node
I20260808 16:00:08.409215 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:08.410580 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:08.411815 17020 hybrid_clock.cc:648] HybridClock initialized: now 1786204808411777 us; error 49 us; skew 500 ppm
I20260808 16:00:08.412154 17020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:08 dist-test-slave-srgv krb5kdc[17007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:08.415540 17020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:08.416898 17020 webserver.cc:460] Webserver started at http://127.0.0.1:36427/ using document root <none> and password file <none>
I20260808 16:00:08.417620 17020 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:08.417771 17020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:08.418087 17020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:08.420934 17020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "806dc0a28aea4b7ab535fd8901db2665"
format_stamp: "Formatted at 2026-08-08 16:00:08 on dist-test-slave-srgv"
server_key: "52e5967075e508263f2584e4ea89c368"
server_key_iv: "d2a7a7fc6289215aa964c7f1ed468abf"
server_key_version: "encryptionkey@0"
I20260808 16:00:08.421743 17020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "806dc0a28aea4b7ab535fd8901db2665"
format_stamp: "Formatted at 2026-08-08 16:00:08 on dist-test-slave-srgv"
server_key: "52e5967075e508263f2584e4ea89c368"
server_key_iv: "d2a7a7fc6289215aa964c7f1ed468abf"
server_key_version: "encryptionkey@0"
I20260808 16:00:08.426476 17020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 16:00:08.429738 17035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:08.430920 17020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:08.431099 17020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "806dc0a28aea4b7ab535fd8901db2665"
format_stamp: "Formatted at 2026-08-08 16:00:08 on dist-test-slave-srgv"
server_key: "52e5967075e508263f2584e4ea89c368"
server_key_iv: "d2a7a7fc6289215aa964c7f1ed468abf"
server_key_version: "encryptionkey@0"
I20260808 16:00:08.431284 17020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:08.451356 17020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:08.455483 17020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:08.455799 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:08.473840 17020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46251
I20260808 16:00:08.473863 17087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46251 every 8 connection(s)
I20260808 16:00:08.475164 17020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:08.478808 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:08.479774 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17020
I20260808 16:00:08.479960 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:08.480391 28664 external_mini_cluster.cc:1428] Setting key 78cfbc5a5fcf220c150faecec0a3e942
I20260808 16:00:08.491957 17088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665: Bootstrap starting.
Aug 08 16:00:08 dist-test-slave-srgv krb5kdc[17007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 16:00:08.495509 17088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:08.496544 17088 log.cc:824] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:08.499346 17088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665: No bootstrap required, opened a new log
I20260808 16:00:08.503268 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:08.482107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47597 (local address 127.0.0.1:46251)
0808 16:00:08.482849 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:08.482863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:08.482905 (+ 42us) server_negotiation.cc:407] Connection header received
0808 16:00:08.484554 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:08.484574 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:08.484937 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:08.485266 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:08.487042 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:08.488260 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:08.490049 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:08.490680 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:08.490811 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:08.495482 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:08.495490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:08.495506 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:08.495543 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:08.498849 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:08.499575 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:08.499582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:08.499589 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:08.499827 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:08.500203 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:08.500207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:08.500211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:08.500883 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:08.501197 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:08.501499 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 16:00:08.501946 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":364,"threads_started":1}
I20260808 16:00:08.506809 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:08.508810 17088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806dc0a28aea4b7ab535fd8901db2665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260808 16:00:08.509227 17088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:08.509342 17088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806dc0a28aea4b7ab535fd8901db2665, State: Initialized, Role: FOLLOWER
I20260808 16:00:08.510027 17088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806dc0a28aea4b7ab535fd8901db2665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260808 16:00:08.510355 17088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:08.510510 17088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:08.510713 17088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:08.513309 17088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806dc0a28aea4b7ab535fd8901db2665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260808 16:00:08.513794 17088 leader_election.cc:302] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 806dc0a28aea4b7ab535fd8901db2665; no voters:
I20260808 16:00:08.514428 17088 leader_election.cc:288] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:08.514524 17098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:08.516888 17088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:08.517695 17098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [term 1 LEADER]: Becoming Leader. State: Replica: 806dc0a28aea4b7ab535fd8901db2665, State: Running, Role: LEADER
I20260808 16:00:08.518294 17098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806dc0a28aea4b7ab535fd8901db2665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260808 16:00:08.521673 17099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "806dc0a28aea4b7ab535fd8901db2665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806dc0a28aea4b7ab535fd8901db2665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } } }
I20260808 16:00:08.522178 17099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:08.523214 17101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806dc0a28aea4b7ab535fd8901db2665. Latest consensus state: current_term: 1 leader_uuid: "806dc0a28aea4b7ab535fd8901db2665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806dc0a28aea4b7ab535fd8901db2665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } } }
I20260808 16:00:08.523590 17101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:08.524531 17104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:08.529793 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:08.512154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56125 (local address 127.0.0.1:46251)
0808 16:00:08.512332 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:08.512338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:08.513944 (+ 1606us) server_negotiation.cc:407] Connection header received
0808 16:00:08.514151 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:08.514156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:08.514228 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:08.514331 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:08.516078 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:08.516874 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:08.518345 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:08.518868 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:08.519001 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:08.522323 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:08.522330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:08.522343 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:08.522371 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:08.525939 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:08.526563 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:08.526569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:08.526573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:08.526756 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:08.527285 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:08.527294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:08.527297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:08.527687 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:08.528998 (+ 1311us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:08.529153 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 16:00:08.529376 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 16:00:08.530769 17104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:08.541368 17104 catalog_manager.cc:1347] Generated new cluster ID: fd3a66cdcca84a969a007fb5e038f3a3
I20260808 16:00:08.541530 17104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:08.567478 17104 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:08.568724 17104 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:08.578537 17104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 806dc0a28aea4b7ab535fd8901db2665: Generated new TSK 0
I20260808 16:00:08.579273 17104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:08.590256 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17020
I20260808 16:00:08.591400 17083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:08.680891 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17020
2026-08-08T16:00:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:08 dist-test-slave-srgv krb5kdc[17121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:08 dist-test-slave-srgv krb5kdc[17121](info): set up 2 sockets
Aug 08 16:00:08 dist-test-slave-srgv krb5kdc[17121](info): commencing operation
krb5kdc: starting...
W20260808 16:00:09.771680 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:09 dist-test-slave-srgv krb5kdc[17121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:10.034570 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:10.359745 17134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:10.360070 17134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:10.360241 17134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:10.367349 17134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:10.367496 17134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:10.367615 17134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:10.367717 17134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:10.367799 17134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:10.377345 17134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:10.378657 17134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:10.379982 17134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:10.391261 17142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:10.391443 17139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:10.391506 17140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:10.392257 17134 server_base.cc:1034] running on GCE node
I20260808 16:00:10.392975 17134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:10.394335 17134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:10.395565 17134 hybrid_clock.cc:648] HybridClock initialized: now 1786204810395533 us; error 46 us; skew 500 ppm
I20260808 16:00:10.395902 17134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:10 dist-test-slave-srgv krb5kdc[17121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:10.398995 17134 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:10.400427 17134 webserver.cc:460] Webserver started at http://127.0.0.1:38655/ using document root <none> and password file <none>
I20260808 16:00:10.401077 17134 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:10.401218 17134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:10.401582 17134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:10.404476 17134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "87b00b5229504ef28e16542807adf9b9"
format_stamp: "Formatted at 2026-08-08 16:00:10 on dist-test-slave-srgv"
server_key: "3cc06724585f14a5b9da87b174e70645"
server_key_iv: "c20de6bb18675f1dc95a14b7cdb29c6c"
server_key_version: "encryptionkey@0"
I20260808 16:00:10.405225 17134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "87b00b5229504ef28e16542807adf9b9"
format_stamp: "Formatted at 2026-08-08 16:00:10 on dist-test-slave-srgv"
server_key: "3cc06724585f14a5b9da87b174e70645"
server_key_iv: "c20de6bb18675f1dc95a14b7cdb29c6c"
server_key_version: "encryptionkey@0"
I20260808 16:00:10.410264 17134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 16:00:10.413648 17149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:10.414911 17134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 16:00:10.415088 17134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "87b00b5229504ef28e16542807adf9b9"
format_stamp: "Formatted at 2026-08-08 16:00:10 on dist-test-slave-srgv"
server_key: "3cc06724585f14a5b9da87b174e70645"
server_key_iv: "c20de6bb18675f1dc95a14b7cdb29c6c"
server_key_version: "encryptionkey@0"
I20260808 16:00:10.415280 17134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:10.436285 17134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:10.440444 17134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:10.440728 17134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:10.458876 17134 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33843
I20260808 16:00:10.458952 17201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33843 every 8 connection(s)
I20260808 16:00:10.460510 17134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:10.464075 17202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:10.468814 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17134
I20260808 16:00:10.469002 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:10.469549 28664 external_mini_cluster.cc:1428] Setting key 16ea4d0e72753e8f93f0ad9b5ecd2c6f
I20260808 16:00:10.474929 17202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9: Bootstrap starting.
I20260808 16:00:10.478299 17202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:10.479406 17202 log.cc:824] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:10.482357 17202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9: No bootstrap required, opened a new log
I20260808 16:00:10.482889 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:10.471483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51173 (local address 127.0.0.1:33843)
0808 16:00:10.472342 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:10.472356 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:10.472395 (+ 39us) server_negotiation.cc:407] Connection header received
0808 16:00:10.474116 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:10.474137 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:10.474509 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:10.474819 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:10.475433 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:10.476716 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:10.478565 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:10.479212 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:10.479335 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:10.480615 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:10.480627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:10.480644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:10.481192 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:10.481258 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:10.481281 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:10.481370 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:10.481654 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 16:00:10.481910 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
I20260808 16:00:10.486335 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:10.489934 17202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b00b5229504ef28e16542807adf9b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33843 } }
I20260808 16:00:10.490283 17202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:10.490439 17202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b00b5229504ef28e16542807adf9b9, State: Initialized, Role: FOLLOWER
I20260808 16:00:10.491135 17202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b00b5229504ef28e16542807adf9b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33843 } }
I20260808 16:00:10.491456 17202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:10.491609 17202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:10.491806 17202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:10.494864 17202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b00b5229504ef28e16542807adf9b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33843 } }
I20260808 16:00:10.495363 17202 leader_election.cc:302] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b00b5229504ef28e16542807adf9b9; no voters:
I20260808 16:00:10.496006 17202 leader_election.cc:288] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:10.496117 17212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:10.497535 17212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [term 1 LEADER]: Becoming Leader. State: Replica: 87b00b5229504ef28e16542807adf9b9, State: Running, Role: LEADER
I20260808 16:00:10.497707 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:10.490569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48103 (local address 127.0.0.1:33843)
0808 16:00:10.490735 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:10.490744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:10.491125 (+ 381us) server_negotiation.cc:407] Connection header received
0808 16:00:10.491285 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:10.491290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:10.491367 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:10.491478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:10.492230 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:10.492940 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:10.494863 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:10.495362 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:10.495494 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:10.496800 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:10.496806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:10.496810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:10.496829 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:10.496869 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:10.496891 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:10.496952 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:10.497198 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 16:00:10.497300 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 16:00:10.498134 17212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b00b5229504ef28e16542807adf9b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33843 } }
I20260808 16:00:10.498432 17202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:10.503015 17213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b00b5229504ef28e16542807adf9b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b00b5229504ef28e16542807adf9b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33843 } } }
I20260808 16:00:10.502995 17214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b00b5229504ef28e16542807adf9b9. Latest consensus state: current_term: 1 leader_uuid: "87b00b5229504ef28e16542807adf9b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b00b5229504ef28e16542807adf9b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33843 } } }
I20260808 16:00:10.503432 17213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:10.503569 17214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:10.509996 17222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:10.513942 17222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:10.521986 17222 catalog_manager.cc:1347] Generated new cluster ID: 70225d90afdb439f8f3eb0d0e4e2413a
I20260808 16:00:10.522121 17222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:10.538232 17222 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:10.540038 17222 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:10.551062 17222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b00b5229504ef28e16542807adf9b9: Generated new TSK 0
I20260808 16:00:10.552044 17222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:10.566212 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17134
W20260808 16:00:10.567683 17197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51173
W20260808 16:00:10.568470 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17134: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:10.568625 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17134
2026-08-08T16:00:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:10 dist-test-slave-srgv krb5kdc[17233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:10 dist-test-slave-srgv krb5kdc[17233](info): set up 2 sockets
Aug 08 16:00:10 dist-test-slave-srgv krb5kdc[17233](info): commencing operation
krb5kdc: starting...
W20260808 16:00:11.651724 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:11 dist-test-slave-srgv krb5kdc[17233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:11.908414 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:12.230486 17246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:12.230770 17246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:12.230896 17246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:12.238073 17246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:12.238204 17246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:12.238310 17246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:12.238420 17246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:12.238519 17246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:12.248550 17246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:12.249933 17246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:12.251228 17246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:12.262912 17253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:12.262933 17255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:12.263715 17252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:12.263722 17246 server_base.cc:1034] running on GCE node
I20260808 16:00:12.264545 17246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:12.265865 17246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:12.267066 17246 hybrid_clock.cc:648] HybridClock initialized: now 1786204812267026 us; error 39 us; skew 500 ppm
I20260808 16:00:12.267410 17246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:12 dist-test-slave-srgv krb5kdc[17233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:12.275544 17246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:12.276995 17246 webserver.cc:460] Webserver started at http://127.0.0.1:42035/ using document root <none> and password file <none>
I20260808 16:00:12.277720 17246 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:12.277860 17246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:12.278146 17246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:12.281028 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "1c106b1093334b2d8f9a1e7a50bb6e66"
format_stamp: "Formatted at 2026-08-08 16:00:12 on dist-test-slave-srgv"
server_key: "821edfc1bb9c2e6d076f557188b7eb8d"
server_key_iv: "22e954cadac89c5b30e280171fcc685b"
server_key_version: "encryptionkey@0"
I20260808 16:00:12.281824 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "1c106b1093334b2d8f9a1e7a50bb6e66"
format_stamp: "Formatted at 2026-08-08 16:00:12 on dist-test-slave-srgv"
server_key: "821edfc1bb9c2e6d076f557188b7eb8d"
server_key_iv: "22e954cadac89c5b30e280171fcc685b"
server_key_version: "encryptionkey@0"
I20260808 16:00:12.286774 17246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 16:00:12.290128 17262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:12.291409 17246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 16:00:12.291590 17246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "1c106b1093334b2d8f9a1e7a50bb6e66"
format_stamp: "Formatted at 2026-08-08 16:00:12 on dist-test-slave-srgv"
server_key: "821edfc1bb9c2e6d076f557188b7eb8d"
server_key_iv: "22e954cadac89c5b30e280171fcc685b"
server_key_version: "encryptionkey@0"
I20260808 16:00:12.291793 17246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:12.297592 17246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:12.301254 17246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:12.301590 17246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:12.319591 17246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39295
I20260808 16:00:12.319638 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39295 every 8 connection(s)
I20260808 16:00:12.320945 17246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:12.324467 17314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:12.331367 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17246
I20260808 16:00:12.331542 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:12.332005 28664 external_mini_cluster.cc:1428] Setting key a834f5eb91b604472d457f5ba29dc1a7
I20260808 16:00:12.334872 17314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66: Bootstrap starting.
I20260808 16:00:12.338929 17314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:12.340288 17314 log.cc:824] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:12.341039 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:12.333855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49229 (local address 127.0.0.1:39295)
0808 16:00:12.334720 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:12.334735 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:12.334785 (+ 50us) server_negotiation.cc:407] Connection header received
0808 16:00:12.337215 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:12.337243 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:12.337708 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:12.338135 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:12.339114 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:12.339125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:12.339140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:12.339157 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:12.339229 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:12.339265 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:12.339465 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:12.339586 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 16:00:12.339795 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":342,"threads_started":1}
I20260808 16:00:12.343109 17314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66: No bootstrap required, opened a new log
I20260808 16:00:12.344573 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:12.351197 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:12.348756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33099 (local address 127.0.0.1:39295)
0808 16:00:12.348945 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:12.348954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:12.349469 (+ 515us) server_negotiation.cc:407] Connection header received
0808 16:00:12.349602 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:12.349606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:12.349647 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:12.349732 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:12.350591 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:12.350594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:12.350597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:12.350610 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:12.350647 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:12.350660 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:12.350758 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:12.350880 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 16:00:12.350921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 16:00:12.351676 17314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260808 16:00:12.352035 17314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:12.352162 17314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c106b1093334b2d8f9a1e7a50bb6e66, State: Initialized, Role: FOLLOWER
I20260808 16:00:12.352705 17314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260808 16:00:12.352942 17314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:12.353088 17314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:12.353277 17314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:12.355710 17314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260808 16:00:12.356196 17314 leader_election.cc:302] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c106b1093334b2d8f9a1e7a50bb6e66; no voters:
I20260808 16:00:12.356719 17314 leader_election.cc:288] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:12.356848 17324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:12.357513 17324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [term 1 LEADER]: Becoming Leader. State: Replica: 1c106b1093334b2d8f9a1e7a50bb6e66, State: Running, Role: LEADER
I20260808 16:00:12.358057 17324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260808 16:00:12.358739 17314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:12.360708 17325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } } }
I20260808 16:00:12.361092 17325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:12.361882 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c106b1093334b2d8f9a1e7a50bb6e66. Latest consensus state: current_term: 1 leader_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c106b1093334b2d8f9a1e7a50bb6e66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } } }
I20260808 16:00:12.362231 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:12.369148 17330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:12.374034 17330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:12.382977 17330 catalog_manager.cc:1347] Generated new cluster ID: 163bd301e9304b8798a9a55355c22b83
I20260808 16:00:12.383075 17330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:12.402017 17330 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:12.402453 17330 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:12.415201 17330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c106b1093334b2d8f9a1e7a50bb6e66: Generated new TSK 0
I20260808 16:00:12.415968 17330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:12.420778 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17246
W20260808 16:00:12.422122 17309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49229
W20260808 16:00:12.423141 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:12.423292 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17246
2026-08-08T16:00:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:12 dist-test-slave-srgv krb5kdc[17345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:12 dist-test-slave-srgv krb5kdc[17345](info): set up 2 sockets
Aug 08 16:00:12 dist-test-slave-srgv krb5kdc[17345](info): commencing operation
krb5kdc: starting...
W20260808 16:00:13.527637 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:13 dist-test-slave-srgv krb5kdc[17345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:13.801079 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:14.132519 17358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:14.132804 17358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:14.132977 17358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:14.140043 17358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:14.140144 17358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:14.140259 17358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:14.140390 17358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:14.140475 17358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:14.150607 17358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:14.151985 17358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:14.153286 17358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:14.165032 17366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:14.165011 17364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:14.165282 17363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:14.165863 17358 server_base.cc:1034] running on GCE node
I20260808 16:00:14.166627 17358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:14.167944 17358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:14.169165 17358 hybrid_clock.cc:648] HybridClock initialized: now 1786204814169102 us; error 52 us; skew 500 ppm
I20260808 16:00:14.169543 17358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:14 dist-test-slave-srgv krb5kdc[17345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:14.172593 17358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:14.174077 17358 webserver.cc:460] Webserver started at http://127.0.0.1:44303/ using document root <none> and password file <none>
I20260808 16:00:14.174698 17358 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:14.174800 17358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:14.175037 17358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:14.177856 17358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "5d348a0878d14e81af88d384029e7c5c"
format_stamp: "Formatted at 2026-08-08 16:00:14 on dist-test-slave-srgv"
server_key: "a3d6b86781f65a1c0f85843accd187cf"
server_key_iv: "4774ce046204c0c7fcc33586befcc959"
server_key_version: "encryptionkey@0"
I20260808 16:00:14.178534 17358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "5d348a0878d14e81af88d384029e7c5c"
format_stamp: "Formatted at 2026-08-08 16:00:14 on dist-test-slave-srgv"
server_key: "a3d6b86781f65a1c0f85843accd187cf"
server_key_iv: "4774ce046204c0c7fcc33586befcc959"
server_key_version: "encryptionkey@0"
I20260808 16:00:14.183298 17358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 16:00:14.186880 17373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:14.188135 17358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 16:00:14.188337 17358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "5d348a0878d14e81af88d384029e7c5c"
format_stamp: "Formatted at 2026-08-08 16:00:14 on dist-test-slave-srgv"
server_key: "a3d6b86781f65a1c0f85843accd187cf"
server_key_iv: "4774ce046204c0c7fcc33586befcc959"
server_key_version: "encryptionkey@0"
I20260808 16:00:14.188530 17358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:14.211658 17358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:14.215569 17358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:14.215857 17358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:14.234247 17358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44571
I20260808 16:00:14.234288 17425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44571 every 8 connection(s)
I20260808 16:00:14.235606 17358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:14.237012 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17358
I20260808 16:00:14.237210 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:14.237800 28664 external_mini_cluster.cc:1428] Setting key 89fc924dabdc703625afae10e6fbade5
I20260808 16:00:14.240125 17426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 16:00:14 dist-test-slave-srgv krb5kdc[17345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 16:00:14.253105 17426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c: Bootstrap starting.
I20260808 16:00:14.257205 17426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:14.258442 17426 log.cc:824] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:14.261734 17426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c: No bootstrap required, opened a new log
I20260808 16:00:14.261883 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:14.239567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:59519 (local address 127.0.0.1:44571)
0808 16:00:14.240371 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:14.240382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:14.240426 (+ 44us) server_negotiation.cc:407] Connection header received
0808 16:00:14.242098 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:14.242120 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:14.242513 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:14.242845 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:14.244647 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:14.246020 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:14.247819 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:14.248428 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:14.248545 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:14.253288 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:14.253299 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:14.253317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:14.253365 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:14.257143 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:14.258049 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:14.258057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:14.258065 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:14.258314 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:14.258869 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:14.258874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:14.258879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:14.259510 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:14.259798 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:14.260280 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 16:00:14.260593 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":356,"threads_started":1}
I20260808 16:00:14.265625 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:14.269526 17426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d348a0878d14e81af88d384029e7c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44571 } }
I20260808 16:00:14.269989 17426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:14.270147 17426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d348a0878d14e81af88d384029e7c5c, State: Initialized, Role: FOLLOWER
I20260808 16:00:14.270751 17426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d348a0878d14e81af88d384029e7c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44571 } }
I20260808 16:00:14.271004 17426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:14.271155 17426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:14.271323 17426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:14.273443 17426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d348a0878d14e81af88d384029e7c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44571 } }
I20260808 16:00:14.273929 17426 leader_election.cc:302] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d348a0878d14e81af88d384029e7c5c; no voters:
I20260808 16:00:14.274565 17426 leader_election.cc:288] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:14.274672 17436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:14.277029 17426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:14.281545 17436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [term 1 LEADER]: Becoming Leader. State: Replica: 5d348a0878d14e81af88d384029e7c5c, State: Running, Role: LEADER
I20260808 16:00:14.282274 17436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d348a0878d14e81af88d384029e7c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44571 } }
I20260808 16:00:14.284819 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:14.269916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:48729 (local address 127.0.0.1:44571)
0808 16:00:14.270121 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:14.270129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:14.270514 (+ 385us) server_negotiation.cc:407] Connection header received
0808 16:00:14.270714 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:14.270719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:14.270794 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:14.270894 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:14.272395 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:14.273160 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:14.274588 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:14.275454 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:14.275612 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:14.278039 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:14.278045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:14.278050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:14.278088 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:14.281304 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:14.282169 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:14.282173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:14.282176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:14.282374 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:14.283046 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:14.283052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:14.283058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:14.283339 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:14.283657 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:14.284047 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 16:00:14.284409 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 16:00:14.286252 17438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d348a0878d14e81af88d384029e7c5c. Latest consensus state: current_term: 1 leader_uuid: "5d348a0878d14e81af88d384029e7c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d348a0878d14e81af88d384029e7c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44571 } } }
I20260808 16:00:14.286204 17437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d348a0878d14e81af88d384029e7c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d348a0878d14e81af88d384029e7c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44571 } } }
I20260808 16:00:14.286659 17438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:14.286691 17437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:14.287508 17445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:14.294389 17445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:14.302809 17445 catalog_manager.cc:1347] Generated new cluster ID: af15d80b408545aab916d7fad23a41fe
I20260808 16:00:14.302913 17445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:14.337649 17445 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:14.338981 17445 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:14.348948 17445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d348a0878d14e81af88d384029e7c5c: Generated new TSK 0
I20260808 16:00:14.349828 17445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:14.361200 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17358
I20260808 16:00:14.362409 17421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:14.453737 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17358
W20260808 16:00:14.470692 17343 connection.cc:535] client connection to 127.0.0.1:44571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44571): Connection reset by peer (error 104)
2026-08-08T16:00:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:14 dist-test-slave-srgv krb5kdc[17458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:14 dist-test-slave-srgv krb5kdc[17458](info): set up 2 sockets
Aug 08 16:00:14 dist-test-slave-srgv krb5kdc[17458](info): commencing operation
krb5kdc: starting...
W20260808 16:00:15.543615 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:15 dist-test-slave-srgv krb5kdc[17458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:15.798396 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:16.124399 17471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:16.124711 17471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:16.124892 17471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:16.132519 17471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:16.132663 17471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:16.132756 17471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:16.132844 17471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:16.132908 17471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:16.142455 17471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:16.143683 17471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:16.144984 17471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:16.156454 17476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:16.156498 17477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:16.156528 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:16.157819 17471 server_base.cc:1034] running on GCE node
I20260808 16:00:16.158601 17471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:16.159950 17471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:16.161188 17471 hybrid_clock.cc:648] HybridClock initialized: now 1786204816161143 us; error 59 us; skew 500 ppm
I20260808 16:00:16.161595 17471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:16 dist-test-slave-srgv krb5kdc[17458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:16.165737 17471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:16.167104 17471 webserver.cc:460] Webserver started at http://127.0.0.1:37469/ using document root <none> and password file <none>
I20260808 16:00:16.167732 17471 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:16.167867 17471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:16.168143 17471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:16.171056 17471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "b21282d33c1f4d8b83cf271478aa2b55"
format_stamp: "Formatted at 2026-08-08 16:00:16 on dist-test-slave-srgv"
server_key: "c184b25f37d0d2acf4ae85dfd69cdfd2"
server_key_iv: "54cae786ca0d729a91bf1bd080b78f4f"
server_key_version: "encryptionkey@0"
I20260808 16:00:16.171756 17471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "b21282d33c1f4d8b83cf271478aa2b55"
format_stamp: "Formatted at 2026-08-08 16:00:16 on dist-test-slave-srgv"
server_key: "c184b25f37d0d2acf4ae85dfd69cdfd2"
server_key_iv: "54cae786ca0d729a91bf1bd080b78f4f"
server_key_version: "encryptionkey@0"
I20260808 16:00:16.176681 17471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 16:00:16.179834 17486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:16.181015 17471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 16:00:16.181181 17471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "b21282d33c1f4d8b83cf271478aa2b55"
format_stamp: "Formatted at 2026-08-08 16:00:16 on dist-test-slave-srgv"
server_key: "c184b25f37d0d2acf4ae85dfd69cdfd2"
server_key_iv: "54cae786ca0d729a91bf1bd080b78f4f"
server_key_version: "encryptionkey@0"
I20260808 16:00:16.181368 17471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:16.194133 17471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:16.198370 17471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:16.198635 17471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:16.216238 17471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42927
I20260808 16:00:16.216281 17538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42927 every 8 connection(s)
I20260808 16:00:16.217613 17471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:16.220937 17539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:16.227546 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17471
I20260808 16:00:16.227736 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:16.228158 28664 external_mini_cluster.cc:1428] Setting key ebae98751dfaf886de84aff5fcb6f5f8
I20260808 16:00:16.231383 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55: Bootstrap starting.
I20260808 16:00:16.235294 17539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:16.236495 17539 log.cc:824] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:16.239447 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55: No bootstrap required, opened a new log
I20260808 16:00:16.243160 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:16.229910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:33883 (local address 127.0.0.1:42927)
0808 16:00:16.230701 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:16.230723 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:16.230785 (+ 62us) server_negotiation.cc:407] Connection header received
0808 16:00:16.233150 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:16.233179 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:16.233736 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:16.234133 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:16.234630 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:16.236609 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:16.238269 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:16.239237 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:16.239402 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:16.240502 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:16.240512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:16.240528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:16.241200 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:16.241289 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:16.241328 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:16.241505 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:16.241769 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 16:00:16.241928 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":322,"threads_started":1}
I20260808 16:00:16.246668 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:16.248051 17539 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21282d33c1f4d8b83cf271478aa2b55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260808 16:00:16.248377 17539 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:16.248471 17539 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21282d33c1f4d8b83cf271478aa2b55, State: Initialized, Role: FOLLOWER
I20260808 16:00:16.249058 17539 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21282d33c1f4d8b83cf271478aa2b55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260808 16:00:16.249300 17539 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:16.249502 17539 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:16.249715 17539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:16.252283 17539 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21282d33c1f4d8b83cf271478aa2b55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260808 16:00:16.252825 17539 leader_election.cc:302] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21282d33c1f4d8b83cf271478aa2b55; no voters:
I20260808 16:00:16.253525 17539 leader_election.cc:288] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:16.253687 17549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:16.254930 17549 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [term 1 LEADER]: Becoming Leader. State: Replica: b21282d33c1f4d8b83cf271478aa2b55, State: Running, Role: LEADER
I20260808 16:00:16.255605 17539 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:16.255537 17549 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21282d33c1f4d8b83cf271478aa2b55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260808 16:00:16.259195 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:16.251136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:36393 (local address 127.0.0.1:42927)
0808 16:00:16.251295 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:16.251301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:16.251636 (+ 335us) server_negotiation.cc:407] Connection header received
0808 16:00:16.251745 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:16.251750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:16.251813 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:16.251896 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:16.252316 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:16.253481 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:16.255557 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:16.256221 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:16.256341 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:16.257826 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:16.257834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:16.257843 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:16.257869 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:16.257927 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:16.257953 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:16.258089 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:16.258641 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 16:00:16.258716 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":62}
I20260808 16:00:16.263357 17550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21282d33c1f4d8b83cf271478aa2b55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21282d33c1f4d8b83cf271478aa2b55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } } }
I20260808 16:00:16.263332 17551 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21282d33c1f4d8b83cf271478aa2b55. Latest consensus state: current_term: 1 leader_uuid: "b21282d33c1f4d8b83cf271478aa2b55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21282d33c1f4d8b83cf271478aa2b55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } } }
I20260808 16:00:16.263792 17550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:16.263816 17551 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:16.265172 17561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:16.269064 17561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:16.276700 17561 catalog_manager.cc:1347] Generated new cluster ID: d933f24f2f2645fbbf18e0580c321e0c
I20260808 16:00:16.276822 17561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:16.299425 17561 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:16.301175 17561 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:16.320750 17561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21282d33c1f4d8b83cf271478aa2b55: Generated new TSK 0
I20260808 16:00:16.321748 17561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:16.327445 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17471
W20260808 16:00:16.328382 17534 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:33883
W20260808 16:00:16.329027 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17471: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:16.329149 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17471
W20260808 16:00:16.346134 17456 connection.cc:535] client connection to 127.0.0.1:42927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42927): Connection reset by peer (error 104)
2026-08-08T16:00:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:16 dist-test-slave-srgv krb5kdc[17570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:16 dist-test-slave-srgv krb5kdc[17570](info): set up 2 sockets
Aug 08 16:00:16 dist-test-slave-srgv krb5kdc[17570](info): commencing operation
krb5kdc: starting...
W20260808 16:00:17.435743 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:17 dist-test-slave-srgv krb5kdc[17570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:17.689990 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:18.014762 17583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:18.015038 17583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:18.015126 17583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:18.022038 17583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:18.022130 17583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:18.022181 17583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:18.022238 17583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:18.022279 17583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:18.031507 17583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:18.032614 17583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:18.033833 17583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:18.045128 17588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:18.045071 17589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:18.045109 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:18.046222 17583 server_base.cc:1034] running on GCE node
I20260808 16:00:18.047041 17583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:18.048347 17583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:18.049579 17583 hybrid_clock.cc:648] HybridClock initialized: now 1786204818049516 us; error 61 us; skew 500 ppm
I20260808 16:00:18.049928 17583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:18 dist-test-slave-srgv krb5kdc[17570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:18.058475 17583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:18.059913 17583 webserver.cc:460] Webserver started at http://127.0.0.1:43893/ using document root <none> and password file <none>
I20260808 16:00:18.060621 17583 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:18.060772 17583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:18.061066 17583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:18.064131 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "efc4309de0e3483b9e80e2ba10b6af89"
format_stamp: "Formatted at 2026-08-08 16:00:18 on dist-test-slave-srgv"
server_key: "f1b2b96287a44f6b34f3d11cc20bce64"
server_key_iv: "084739a284b1d723fb9e855a1f8373d0"
server_key_version: "encryptionkey@0"
I20260808 16:00:18.064890 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "efc4309de0e3483b9e80e2ba10b6af89"
format_stamp: "Formatted at 2026-08-08 16:00:18 on dist-test-slave-srgv"
server_key: "f1b2b96287a44f6b34f3d11cc20bce64"
server_key_iv: "084739a284b1d723fb9e855a1f8373d0"
server_key_version: "encryptionkey@0"
I20260808 16:00:18.070048 17583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 16:00:18.073391 17598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:18.074740 17583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 16:00:18.074922 17583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "efc4309de0e3483b9e80e2ba10b6af89"
format_stamp: "Formatted at 2026-08-08 16:00:18 on dist-test-slave-srgv"
server_key: "f1b2b96287a44f6b34f3d11cc20bce64"
server_key_iv: "084739a284b1d723fb9e855a1f8373d0"
server_key_version: "encryptionkey@0"
I20260808 16:00:18.075115 17583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:18.080623 17583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:18.084340 17583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:18.084617 17583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:18.102519 17583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42715
I20260808 16:00:18.102598 17649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42715 every 8 connection(s)
I20260808 16:00:18.103859 17583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:18.104832 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17583
I20260808 16:00:18.105020 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:18.105571 28664 external_mini_cluster.cc:1428] Setting key db989348ad8e65411ed9fb36e821e44e
I20260808 16:00:18.108031 17650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:18.113301 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:18.107404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:57069 (local address 127.0.0.1:42715)
0808 16:00:18.108253 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:18.108265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:18.108307 (+ 42us) server_negotiation.cc:407] Connection header received
0808 16:00:18.109983 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:18.110003 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:18.110345 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:18.110684 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:18.111742 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:18.111748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:18.111758 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:18.111778 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:18.111864 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:18.111901 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:18.112067 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:18.112189 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 16:00:18.112362 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":394,"threads_started":1}
I20260808 16:00:18.116962 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:18.119774 17650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89: Bootstrap starting.
I20260808 16:00:18.123126 17650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:18.123879 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:18.121338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:35909 (local address 127.0.0.1:42715)
0808 16:00:18.121507 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:18.121510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:18.121986 (+ 476us) server_negotiation.cc:407] Connection header received
0808 16:00:18.122177 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:18.122182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:18.122234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:18.122330 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:18.123226 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:18.123232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:18.123239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:18.123255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:18.123299 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:18.123316 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:18.123439 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:18.123546 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 16:00:18.123602 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 16:00:18.124393 17650 log.cc:824] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:18.127023 17650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89: No bootstrap required, opened a new log
I20260808 16:00:18.134191 17650 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc4309de0e3483b9e80e2ba10b6af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260808 16:00:18.134496 17650 raft_consensus.cc:374] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:18.134600 17650 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc4309de0e3483b9e80e2ba10b6af89, State: Initialized, Role: FOLLOWER
I20260808 16:00:18.135259 17650 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc4309de0e3483b9e80e2ba10b6af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260808 16:00:18.135519 17650 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:18.135632 17650 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:18.135753 17650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:18.137804 17650 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc4309de0e3483b9e80e2ba10b6af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260808 16:00:18.138195 17650 leader_election.cc:302] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc4309de0e3483b9e80e2ba10b6af89; no voters:
I20260808 16:00:18.138708 17650 leader_election.cc:288] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:18.138814 17660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:18.139843 17660 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [term 1 LEADER]: Becoming Leader. State: Replica: efc4309de0e3483b9e80e2ba10b6af89, State: Running, Role: LEADER
I20260808 16:00:18.140365 17660 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc4309de0e3483b9e80e2ba10b6af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260808 16:00:18.140739 17650 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:18.143098 17662 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc4309de0e3483b9e80e2ba10b6af89. Latest consensus state: current_term: 1 leader_uuid: "efc4309de0e3483b9e80e2ba10b6af89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc4309de0e3483b9e80e2ba10b6af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } } }
I20260808 16:00:18.143589 17662 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:18.143126 17661 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efc4309de0e3483b9e80e2ba10b6af89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc4309de0e3483b9e80e2ba10b6af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } } }
I20260808 16:00:18.143831 17661 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:18.145910 17668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:18.150975 17668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:18.159286 17668 catalog_manager.cc:1347] Generated new cluster ID: 0922da03c9d840659017119006e8a302
I20260808 16:00:18.159392 17668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:18.171469 17668 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:18.171846 17668 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:18.180420 17668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efc4309de0e3483b9e80e2ba10b6af89: Generated new TSK 0
I20260808 16:00:18.181102 17668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:18.189848 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17583
W20260808 16:00:18.191465 17645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:57069
W20260808 16:00:18.192154 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:18.192314 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17583
2026-08-08T16:00:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:18 dist-test-slave-srgv krb5kdc[17682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:18 dist-test-slave-srgv krb5kdc[17682](info): set up 2 sockets
Aug 08 16:00:18 dist-test-slave-srgv krb5kdc[17682](info): commencing operation
krb5kdc: starting...
W20260808 16:00:19.295861 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:19 dist-test-slave-srgv krb5kdc[17682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:19.548694 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:19.874948 17695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:19.875221 17695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:19.875391 17695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:19.882658 17695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:19.882798 17695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:19.882901 17695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:19.883005 17695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:19.883087 17695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:19.892920 17695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:19.894277 17695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:19.895535 17695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:19.907289 17703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:19.907259 17700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:19.907325 17701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:19.908033 17695 server_base.cc:1034] running on GCE node
I20260808 16:00:19.908814 17695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:19.910266 17695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:19.911501 17695 hybrid_clock.cc:648] HybridClock initialized: now 1786204819911463 us; error 47 us; skew 500 ppm
I20260808 16:00:19.911837 17695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:19 dist-test-slave-srgv krb5kdc[17682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:19.914949 17695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:19.916445 17695 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
I20260808 16:00:19.917114 17695 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:19.917254 17695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:19.917619 17695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:19.920563 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5"
format_stamp: "Formatted at 2026-08-08 16:00:19 on dist-test-slave-srgv"
server_key: "01e66267048a68a6cec036ad8a2b16bb"
server_key_iv: "024d859684e377d7dd8d78166f4b7597"
server_key_version: "encryptionkey@0"
I20260808 16:00:19.921312 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5"
format_stamp: "Formatted at 2026-08-08 16:00:19 on dist-test-slave-srgv"
server_key: "01e66267048a68a6cec036ad8a2b16bb"
server_key_iv: "024d859684e377d7dd8d78166f4b7597"
server_key_version: "encryptionkey@0"
I20260808 16:00:19.926203 17695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 16:00:19.929556 17710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:19.930785 17695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 16:00:19.930965 17695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5"
format_stamp: "Formatted at 2026-08-08 16:00:19 on dist-test-slave-srgv"
server_key: "01e66267048a68a6cec036ad8a2b16bb"
server_key_iv: "024d859684e377d7dd8d78166f4b7597"
server_key_version: "encryptionkey@0"
I20260808 16:00:19.931162 17695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:19.952461 17695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:19.958408 17695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:19.958729 17695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:19.979697 17695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40613
I20260808 16:00:19.979734 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40613 every 8 connection(s)
I20260808 16:00:19.981020 17695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:19.982710 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17695
I20260808 16:00:19.982882 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:19.983436 28664 external_mini_cluster.cc:1428] Setting key 2bcc484d2ea0428ce4ea1c87a0013c91
I20260808 16:00:19.985484 17762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:19.996119 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5: Bootstrap starting.
Aug 08 16:00:19 dist-test-slave-srgv krb5kdc[17682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 16:00:19.999814 17762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:20.000864 17762 log.cc:824] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:20.003912 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5: No bootstrap required, opened a new log
I20260808 16:00:20.005470 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:19.985227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:45413 (local address 127.0.0.1:40613)
0808 16:00:19.986122 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:19.986134 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:19.986177 (+ 43us) server_negotiation.cc:407] Connection header received
0808 16:00:19.987732 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:19.987752 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:19.988133 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:19.988543 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:19.990073 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:19.991374 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:19.992655 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:19.993362 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:19.993613 (+ 251us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:19.998194 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:19.998201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:19.998214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:19.998251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:20.001223 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:20.002204 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:20.002210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:20.002216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:20.002375 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:20.002865 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:20.002869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:20.002870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:20.003317 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:20.003558 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:20.004143 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 16:00:20.004488 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":455,"threads_started":1}
I20260808 16:00:20.009280 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:20.013031 17762 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260808 16:00:20.013612 17762 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:20.013729 17762 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d2aa5a304c4928a5b9daa220fb8cf5, State: Initialized, Role: FOLLOWER
I20260808 16:00:20.014361 17762 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260808 16:00:20.014856 17762 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:20.015018 17762 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:20.015232 17762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:20.018390 17762 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260808 16:00:20.018882 17762 leader_election.cc:302] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3d2aa5a304c4928a5b9daa220fb8cf5; no voters:
I20260808 16:00:20.019474 17762 leader_election.cc:288] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:20.019609 17772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:20.020593 17772 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [term 1 LEADER]: Becoming Leader. State: Replica: e3d2aa5a304c4928a5b9daa220fb8cf5, State: Running, Role: LEADER
I20260808 16:00:20.021282 17772 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260808 16:00:20.021669 17762 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:20.024140 17773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260808 16:00:20.024530 17773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:20.025466 17774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d2aa5a304c4928a5b9daa220fb8cf5. Latest consensus state: current_term: 1 leader_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d2aa5a304c4928a5b9daa220fb8cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260808 16:00:20.025871 17774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:20.029227 17777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:20.034212 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:20.014777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:44953 (local address 127.0.0.1:40613)
0808 16:00:20.014991 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:20.015000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:20.015494 (+ 494us) server_negotiation.cc:407] Connection header received
0808 16:00:20.015688 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:20.015691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:20.015749 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:20.015827 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:20.017738 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:20.018579 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:20.020526 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:20.021241 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:20.021368 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:20.025019 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:20.025029 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:20.025036 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:20.025089 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:20.030160 (+ 5071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:20.031192 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:20.031201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:20.031207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:20.031490 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:20.032152 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:20.032157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:20.032160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:20.032551 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:20.032841 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:20.033621 (+ 780us) server_negotiation.cc:299] Negotiation successful
0808 16:00:20.033833 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 16:00:20.035812 17777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:20.045008 17777 catalog_manager.cc:1347] Generated new cluster ID: 3e756a39d7534ec2adde6a4d9b41c386
I20260808 16:00:20.045169 17777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:20.063037 17777 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:20.064946 17777 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:20.071430 17777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3d2aa5a304c4928a5b9daa220fb8cf5: Generated new TSK 0
I20260808 16:00:20.072495 17777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:20.079107 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17695
I20260808 16:00:20.080598 17757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:20.169773 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17695
W20260808 16:00:20.185828 17680 connection.cc:535] client connection to 127.0.0.1:40613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40613): Connection reset by peer (error 104)
2026-08-08T16:00:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:20 dist-test-slave-srgv krb5kdc[17794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:20 dist-test-slave-srgv krb5kdc[17794](info): set up 2 sockets
Aug 08 16:00:20 dist-test-slave-srgv krb5kdc[17794](info): commencing operation
krb5kdc: starting...
W20260808 16:00:21.259680 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:21 dist-test-slave-srgv krb5kdc[17794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:21.524271 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:21.846232 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:21.846488 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:21.846673 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:21.854791 17807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:21.854921 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:21.855002 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:21.855111 17807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:21.855192 17807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:21.864823 17807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:21.866104 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:21.867367 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:21.879076 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:21.879173 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:21.879149 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:21.880328 17807 server_base.cc:1034] running on GCE node
I20260808 16:00:21.881069 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:21.882373 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:21.883567 17807 hybrid_clock.cc:648] HybridClock initialized: now 1786204821883527 us; error 34 us; skew 500 ppm
I20260808 16:00:21.883901 17807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:21 dist-test-slave-srgv krb5kdc[17794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:21.886833 17807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:21.888199 17807 webserver.cc:460] Webserver started at http://127.0.0.1:42637/ using document root <none> and password file <none>
I20260808 16:00:21.888876 17807 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:21.889019 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:21.889317 17807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:21.892167 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "a64e55df05a649e5a9f5ee3a3d0e9006"
format_stamp: "Formatted at 2026-08-08 16:00:21 on dist-test-slave-srgv"
server_key: "f8f1c9ce4737a65bd4554a271df7a00e"
server_key_iv: "cd5c2444cdd20059ed0ffc4bd17ec377"
server_key_version: "encryptionkey@0"
I20260808 16:00:21.892910 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "a64e55df05a649e5a9f5ee3a3d0e9006"
format_stamp: "Formatted at 2026-08-08 16:00:21 on dist-test-slave-srgv"
server_key: "f8f1c9ce4737a65bd4554a271df7a00e"
server_key_iv: "cd5c2444cdd20059ed0ffc4bd17ec377"
server_key_version: "encryptionkey@0"
I20260808 16:00:21.897791 17807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260808 16:00:21.901041 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:21.902303 17807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:21.902474 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "a64e55df05a649e5a9f5ee3a3d0e9006"
format_stamp: "Formatted at 2026-08-08 16:00:21 on dist-test-slave-srgv"
server_key: "f8f1c9ce4737a65bd4554a271df7a00e"
server_key_iv: "cd5c2444cdd20059ed0ffc4bd17ec377"
server_key_version: "encryptionkey@0"
I20260808 16:00:21.902659 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:21.918442 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:21.922093 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:21.922366 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:21.940382 17807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40511
I20260808 16:00:21.940414 17874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40511 every 8 connection(s)
I20260808 16:00:21.941718 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:21.945176 17876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:21.948635 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17807
I20260808 16:00:21.948839 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:21.949355 28664 external_mini_cluster.cc:1428] Setting key d2dbe3e46d1d8c71fe7f600d37dd8a24
I20260808 16:00:21.957551 17876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006: Bootstrap starting.
I20260808 16:00:21.960451 17876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:21.961552 17876 log.cc:824] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:21.962598 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:21.951865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:58661 (local address 127.0.0.1:40511)
0808 16:00:21.952547 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:21.952558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:21.952594 (+ 36us) server_negotiation.cc:407] Connection header received
0808 16:00:21.954273 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:21.954293 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:21.954679 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:21.955049 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:21.955592 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:21.956861 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:21.958309 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:21.958963 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:21.959169 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:21.960353 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:21.960364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:21.960377 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:21.960946 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:21.961013 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:21.961036 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:21.961195 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:21.961491 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 16:00:21.961633 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260808 16:00:21.964601 17876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006: No bootstrap required, opened a new log
I20260808 16:00:21.966429 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:21.973675 17876 raft_consensus.cc:348] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40511 } }
I20260808 16:00:21.974144 17876 raft_consensus.cc:374] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:21.974275 17876 raft_consensus.cc:729] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64e55df05a649e5a9f5ee3a3d0e9006, State: Initialized, Role: FOLLOWER
I20260808 16:00:21.974900 17876 consensus_queue.cc:260] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40511 } }
I20260808 16:00:21.975165 17876 raft_consensus.cc:388] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:21.975301 17876 raft_consensus.cc:482] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:21.975469 17876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:21.978266 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:21.971385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:38801 (local address 127.0.0.1:40511)
0808 16:00:21.971529 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:21.971533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:21.972039 (+ 506us) server_negotiation.cc:407] Connection header received
0808 16:00:21.972238 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:21.972241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:21.972302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:21.972384 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:21.972954 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:21.973728 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:21.975720 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:21.976135 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:21.976240 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:21.977382 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:21.977387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:21.977390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:21.977497 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:21.977561 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:21.977580 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:21.977672 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:21.977894 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 16:00:21.977933 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 16:00:21.978564 17876 raft_consensus.cc:504] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40511 } }
I20260808 16:00:21.979065 17876 leader_election.cc:302] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a64e55df05a649e5a9f5ee3a3d0e9006; no voters:
I20260808 16:00:21.979739 17876 leader_election.cc:288] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:21.979861 17886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:21.980531 17886 raft_consensus.cc:686] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [term 1 LEADER]: Becoming Leader. State: Replica: a64e55df05a649e5a9f5ee3a3d0e9006, State: Running, Role: LEADER
I20260808 16:00:21.981091 17886 consensus_queue.cc:237] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40511 } }
I20260808 16:00:21.981958 17876 sys_catalog.cc:564] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:21.983964 17887 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40511 } } }
I20260808 16:00:21.984097 17888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64e55df05a649e5a9f5ee3a3d0e9006. Latest consensus state: current_term: 1 leader_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e55df05a649e5a9f5ee3a3d0e9006" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40511 } } }
I20260808 16:00:21.984409 17887 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:21.984508 17888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:21.986891 17891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:21.991999 17891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:22.001513 17891 catalog_manager.cc:1347] Generated new cluster ID: 66312f6a34484016aec9aa890cbf0cba
I20260808 16:00:22.001633 17891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:22.015905 17891 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:22.017084 17891 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:22.028062 17891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a64e55df05a649e5a9f5ee3a3d0e9006: Generated new TSK 0
I20260808 16:00:22.028806 17891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:22.037312 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17807
W20260808 16:00:22.038383 17870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:58661
W20260808 16:00:22.039076 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:22.039230 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17807
W20260808 16:00:22.055553 17792 connection.cc:535] client connection to 127.0.0.1:40511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40511): Connection reset by peer (error 104)
2026-08-08T16:00:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:22 dist-test-slave-srgv krb5kdc[17907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:22 dist-test-slave-srgv krb5kdc[17907](info): set up 2 sockets
Aug 08 16:00:22 dist-test-slave-srgv krb5kdc[17907](info): commencing operation
krb5kdc: starting...
W20260808 16:00:23.131796 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:23 dist-test-slave-srgv krb5kdc[17907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 16:00:23.386449 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:23.727116 17920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:23.727394 17920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:23.727558 17920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:23.734652 17920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:23.734751 17920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:23.734860 17920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:23.734961 17920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:23.735040 17920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:23.744647 17920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:23.745916 17920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:23.747195 17920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:23.758786 17926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:23.758821 17928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:23.758872 17925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:23.759354 17920 server_base.cc:1034] running on GCE node
I20260808 16:00:23.760144 17920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:23.761528 17920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:23.762763 17920 hybrid_clock.cc:648] HybridClock initialized: now 1786204823762680 us; error 71 us; skew 500 ppm
I20260808 16:00:23.763126 17920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:23 dist-test-slave-srgv krb5kdc[17907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:23.766487 17920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 16:00:23.767828 17920 webserver.cc:460] Webserver started at http://127.0.0.1:40043/ using document root <none> and password file <none>
I20260808 16:00:23.768458 17920 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:23.768560 17920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:23.768810 17920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:23.771737 17920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "49e060bc800c4c8191e60e87577dcb0d"
format_stamp: "Formatted at 2026-08-08 16:00:23 on dist-test-slave-srgv"
server_key: "741398e1014b583b7139de6f5d941633"
server_key_iv: "7b23e941926128de9b68df6fd40102d2"
server_key_version: "encryptionkey@0"
I20260808 16:00:23.772418 17920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "49e060bc800c4c8191e60e87577dcb0d"
format_stamp: "Formatted at 2026-08-08 16:00:23 on dist-test-slave-srgv"
server_key: "741398e1014b583b7139de6f5d941633"
server_key_iv: "7b23e941926128de9b68df6fd40102d2"
server_key_version: "encryptionkey@0"
I20260808 16:00:23.777379 17920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 16:00:23.780802 17935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:23.782131 17920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:23.782269 17920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "49e060bc800c4c8191e60e87577dcb0d"
format_stamp: "Formatted at 2026-08-08 16:00:23 on dist-test-slave-srgv"
server_key: "741398e1014b583b7139de6f5d941633"
server_key_iv: "7b23e941926128de9b68df6fd40102d2"
server_key_version: "encryptionkey@0"
I20260808 16:00:23.782431 17920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:23.787808 17920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:23.791982 17920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:23.792224 17920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:23.810030 17920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39589
I20260808 16:00:23.810120 17987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39589 every 8 connection(s)
I20260808 16:00:23.811407 17920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:23.814981 17988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:23.820402 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 17920
I20260808 16:00:23.820580 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:23.821125 28664 external_mini_cluster.cc:1428] Setting key 5e39b2cb2b6172115b13f44577be3c19
I20260808 16:00:23.825805 17988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d: Bootstrap starting.
I20260808 16:00:23.828773 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:23.822938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:59005 (local address 127.0.0.1:39589)
0808 16:00:23.823759 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:23.823770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:23.823810 (+ 40us) server_negotiation.cc:407] Connection header received
0808 16:00:23.825543 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:23.825573 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:23.825946 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:23.826293 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:23.827299 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:23.827305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:23.827316 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:23.827329 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:23.827390 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:23.827409 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:23.827577 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:23.827713 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 16:00:23.827898 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":396,"threads_started":1}
I20260808 16:00:23.829324 17988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:23.830385 17988 log.cc:824] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:23.832510 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:23.833508 17988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d: No bootstrap required, opened a new log
I20260808 16:00:23.839933 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:23.837630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:60025 (local address 127.0.0.1:39589)
0808 16:00:23.837776 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:23.837780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:23.838153 (+ 373us) server_negotiation.cc:407] Connection header received
0808 16:00:23.838316 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:23.838320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:23.838358 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:23.838424 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:23.839307 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:23.839314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:23.839317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:23.839330 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:23.839363 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:23.839375 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:23.839469 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:23.839588 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 16:00:23.839630 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 16:00:23.843137 17988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e060bc800c4c8191e60e87577dcb0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39589 } }
I20260808 16:00:23.843485 17988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:23.843578 17988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e060bc800c4c8191e60e87577dcb0d, State: Initialized, Role: FOLLOWER
I20260808 16:00:23.844139 17988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e060bc800c4c8191e60e87577dcb0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39589 } }
I20260808 16:00:23.844389 17988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:23.844540 17988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:23.844710 17988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:23.846892 17988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e060bc800c4c8191e60e87577dcb0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39589 } }
I20260808 16:00:23.847353 17988 leader_election.cc:302] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49e060bc800c4c8191e60e87577dcb0d; no voters:
I20260808 16:00:23.847914 17988 leader_election.cc:288] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:23.848055 17998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:23.848738 17998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [term 1 LEADER]: Becoming Leader. State: Replica: 49e060bc800c4c8191e60e87577dcb0d, State: Running, Role: LEADER
I20260808 16:00:23.849260 17998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e060bc800c4c8191e60e87577dcb0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39589 } }
I20260808 16:00:23.850078 17988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:23.852521 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49e060bc800c4c8191e60e87577dcb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e060bc800c4c8191e60e87577dcb0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39589 } } }
I20260808 16:00:23.852715 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49e060bc800c4c8191e60e87577dcb0d. Latest consensus state: current_term: 1 leader_uuid: "49e060bc800c4c8191e60e87577dcb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e060bc800c4c8191e60e87577dcb0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39589 } } }
I20260808 16:00:23.853119 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:23.853116 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:23.854712 18004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:23.861896 18004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:23.870471 18004 catalog_manager.cc:1347] Generated new cluster ID: 749f7e292cc14a65aa41c19335fa1ff1
I20260808 16:00:23.870579 18004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:23.896150 18004 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:23.896591 18004 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:23.904110 18004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49e060bc800c4c8191e60e87577dcb0d: Generated new TSK 0
I20260808 16:00:23.904922 18004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:23.909991 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17920
W20260808 16:00:23.911123 17983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:59005
W20260808 16:00:23.911825 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 17920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:23.911993 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 17920
2026-08-08T16:00:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:24 dist-test-slave-srgv krb5kdc[18019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:24 dist-test-slave-srgv krb5kdc[18019](info): set up 2 sockets
Aug 08 16:00:24 dist-test-slave-srgv krb5kdc[18019](info): commencing operation
krb5kdc: starting...
W20260808 16:00:24.999671 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:25 dist-test-slave-srgv krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:25.241092 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37673
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:25.561805 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:25.562071 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:25.562193 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:25.569051 18033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:25.569154 18033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:25.569197 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:25.569249 18033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:25.569285 18033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:25.579021 18033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:25.580241 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:25.581481 18033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:25.594377 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:25.594424 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:25.594345 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:25.595847 18033 server_base.cc:1034] running on GCE node
I20260808 16:00:25.596553 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:25.597934 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:25.599226 18033 hybrid_clock.cc:648] HybridClock initialized: now 1786204825599163 us; error 55 us; skew 500 ppm
I20260808 16:00:25.599539 18033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:25 dist-test-slave-srgv krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:25.602698 18033 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:25.604105 18033 webserver.cc:460] Webserver started at http://127.27.254.62:39587/ using document root <none> and password file <none>
I20260808 16:00:25.604746 18033 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:25.604841 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:25.605072 18033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:25.607952 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "f00cdead02964f36a4e663c5f02db346"
format_stamp: "Formatted at 2026-08-08 16:00:25 on dist-test-slave-srgv"
server_key: "c6ce8d22badc329d97562ea52cafd50f"
server_key_iv: "f6b1b2f6017e609f3cc7373a5dcaf8cc"
server_key_version: "encryptionkey@0"
I20260808 16:00:25.608631 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "f00cdead02964f36a4e663c5f02db346"
format_stamp: "Formatted at 2026-08-08 16:00:25 on dist-test-slave-srgv"
server_key: "c6ce8d22badc329d97562ea52cafd50f"
server_key_iv: "f6b1b2f6017e609f3cc7373a5dcaf8cc"
server_key_version: "encryptionkey@0"
I20260808 16:00:25.613637 18033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 16:00:25.616849 18048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:25.618193 18033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:25.618377 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "f00cdead02964f36a4e663c5f02db346"
format_stamp: "Formatted at 2026-08-08 16:00:25 on dist-test-slave-srgv"
server_key: "c6ce8d22badc329d97562ea52cafd50f"
server_key_iv: "f6b1b2f6017e609f3cc7373a5dcaf8cc"
server_key_version: "encryptionkey@0"
I20260808 16:00:25.618566 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:25.637029 18033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:25.641597 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:25.641896 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:25.660373 18033 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:37673
I20260808 16:00:25.660392 18099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:37673 every 8 connection(s)
I20260808 16:00:25.661783 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:25.664227 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18033
I20260808 16:00:25.664407 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:25.664875 28664 external_mini_cluster.cc:1428] Setting key ece4a70890f618b7bd7c048f0685ff25
I20260808 16:00:25.665784 18100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 16:00:25 dist-test-slave-srgv krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 16:00:25.679679 18100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346: Bootstrap starting.
I20260808 16:00:25.683597 18100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:25.684633 18100 log.cc:824] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:25.687554 18100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346: No bootstrap required, opened a new log
I20260808 16:00:25.688577 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:25.666858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49185 (local address 127.27.254.62:37673)
0808 16:00:25.667648 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:25.667662 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:25.667721 (+ 59us) server_negotiation.cc:407] Connection header received
0808 16:00:25.669351 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:25.669378 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:25.669946 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:25.670380 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:25.671747 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:25.673809 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:25.675101 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:25.676153 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:25.676355 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:25.680589 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:25.680599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:25.680618 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:25.680660 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:25.684204 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:25.685061 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:25.685072 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:25.685083 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:25.685333 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:25.685866 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:25.685874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:25.685880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:25.686416 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:25.686772 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:25.687299 (+ 527us) server_negotiation.cc:299] Negotiation successful
0808 16:00:25.687599 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":329,"threads_started":1}
I20260808 16:00:25.692438 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:25.695508 18100 raft_consensus.cc:348] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00cdead02964f36a4e663c5f02db346" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37673 } }
I20260808 16:00:25.695866 18100 raft_consensus.cc:374] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:25.695986 18100 raft_consensus.cc:729] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00cdead02964f36a4e663c5f02db346, State: Initialized, Role: FOLLOWER
I20260808 16:00:25.696586 18100 consensus_queue.cc:260] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00cdead02964f36a4e663c5f02db346" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37673 } }
I20260808 16:00:25.696874 18100 raft_consensus.cc:388] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:25.697058 18100 raft_consensus.cc:482] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:25.697289 18100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:25.699549 18100 raft_consensus.cc:504] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00cdead02964f36a4e663c5f02db346" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37673 } }
I20260808 16:00:25.700062 18100 leader_election.cc:302] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f00cdead02964f36a4e663c5f02db346; no voters:
I20260808 16:00:25.700723 18100 leader_election.cc:288] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:25.700831 18110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:25.702333 18110 raft_consensus.cc:686] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [term 1 LEADER]: Becoming Leader. State: Replica: f00cdead02964f36a4e663c5f02db346, State: Running, Role: LEADER
I20260808 16:00:25.702926 18110 consensus_queue.cc:237] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00cdead02964f36a4e663c5f02db346" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37673 } }
I20260808 16:00:25.703137 18100 sys_catalog.cc:564] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:25.706789 18112 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f00cdead02964f36a4e663c5f02db346. Latest consensus state: current_term: 1 leader_uuid: "f00cdead02964f36a4e663c5f02db346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00cdead02964f36a4e663c5f02db346" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37673 } } }
I20260808 16:00:25.706818 18111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f00cdead02964f36a4e663c5f02db346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00cdead02964f36a4e663c5f02db346" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37673 } } }
I20260808 16:00:25.707505 18112 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:25.707505 18111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:25.711846 18119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:25.714916 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:25.696993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34301 (local address 127.27.254.62:37673)
0808 16:00:25.697159 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:25.697166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:25.697712 (+ 546us) server_negotiation.cc:407] Connection header received
0808 16:00:25.697911 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:25.697916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:25.697994 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:25.698107 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:25.699522 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:25.700411 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:25.702101 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:25.702810 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:25.702934 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:25.706390 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:25.706398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:25.706405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:25.706448 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:25.710901 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:25.711914 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:25.711921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:25.711927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:25.712126 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:25.712712 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:25.712716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:25.712718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:25.713016 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:25.713352 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:25.714338 (+ 986us) server_negotiation.cc:299] Negotiation successful
0808 16:00:25.714541 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 16:00:25.718564 18119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:25.727294 18119 catalog_manager.cc:1347] Generated new cluster ID: 92d1555cacb0433ba051436afe69529e
I20260808 16:00:25.727399 18119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:25.735960 18119 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:25.737142 18119 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:25.747886 18119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f00cdead02964f36a4e663c5f02db346: Generated new TSK 0
I20260808 16:00:25.748700 18119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:25.756704 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18033
I20260808 16:00:25.757853 18095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:25.845125 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18033
W20260808 16:00:25.861634 18017 connection.cc:535] client connection to 127.27.254.62:37673 recv error: Network error: failed to read from TLS socket (remote: 127.27.254.62:37673): Connection reset by peer (error 104)
2026-08-08T16:00:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:25 dist-test-slave-srgv krb5kdc[18132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:25 dist-test-slave-srgv krb5kdc[18132](info): set up 2 sockets
Aug 08 16:00:25 dist-test-slave-srgv krb5kdc[18132](info): commencing operation
krb5kdc: starting...
W20260808 16:00:26.935600 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:27 dist-test-slave-srgv krb5kdc[18132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:27.180122 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46845
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:27.501503 18145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:27.501837 18145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:27.501986 18145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:27.508793 18145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:27.508929 18145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:27.509035 18145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:27.509132 18145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:27.509214 18145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:27.518683 18145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:46845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:27.519933 18145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:27.521194 18145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:27.532804 18150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:27.532682 18153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:27.532756 18151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:27.533609 18145 server_base.cc:1034] running on GCE node
I20260808 16:00:27.534394 18145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:27.535761 18145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:27.536994 18145 hybrid_clock.cc:648] HybridClock initialized: now 1786204827536966 us; error 57 us; skew 500 ppm
I20260808 16:00:27.537341 18145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:27 dist-test-slave-srgv krb5kdc[18132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204827, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:27.540300 18145 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:27.541679 18145 webserver.cc:460] Webserver started at http://127.27.254.62:33481/ using document root <none> and password file <none>
I20260808 16:00:27.542294 18145 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:27.542392 18145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:27.542635 18145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:27.545534 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "9236068e4bc94d24b9047d1e40b21b52"
format_stamp: "Formatted at 2026-08-08 16:00:27 on dist-test-slave-srgv"
server_key: "103630ac1a026d2ecac15652f57b743c"
server_key_iv: "d8ef8883356960b92ecee66cf839867c"
server_key_version: "encryptionkey@0"
I20260808 16:00:27.546262 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "9236068e4bc94d24b9047d1e40b21b52"
format_stamp: "Formatted at 2026-08-08 16:00:27 on dist-test-slave-srgv"
server_key: "103630ac1a026d2ecac15652f57b743c"
server_key_iv: "d8ef8883356960b92ecee66cf839867c"
server_key_version: "encryptionkey@0"
I20260808 16:00:27.551077 18145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 16:00:27.554203 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:27.555433 18145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 16:00:27.555564 18145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "9236068e4bc94d24b9047d1e40b21b52"
format_stamp: "Formatted at 2026-08-08 16:00:27 on dist-test-slave-srgv"
server_key: "103630ac1a026d2ecac15652f57b743c"
server_key_iv: "d8ef8883356960b92ecee66cf839867c"
server_key_version: "encryptionkey@0"
I20260808 16:00:27.555727 18145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:27.572292 18145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:27.576527 18145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:27.576802 18145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:27.594604 18145 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:46845
I20260808 16:00:27.594626 18212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:46845 every 8 connection(s)
I20260808 16:00:27.595870 18145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:27.599306 18213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:27.604832 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18145
I20260808 16:00:27.605008 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:27.605507 28664 external_mini_cluster.cc:1428] Setting key 3a1c1a8630284704e0eb7c78df515e16
I20260808 16:00:27.609628 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52: Bootstrap starting.
I20260808 16:00:27.612510 18213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:27.613520 18213 log.cc:824] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:27.616309 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52: No bootstrap required, opened a new log
I20260808 16:00:27.618546 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:27.607337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46657 (local address 127.27.254.62:46845)
0808 16:00:27.608094 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:27.608106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:27.608143 (+ 37us) server_negotiation.cc:407] Connection header received
0808 16:00:27.609860 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:27.609881 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:27.610244 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:27.610596 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:27.611140 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:27.612330 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:27.613741 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:27.614832 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:27.615016 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:27.616279 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:27.616286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:27.616302 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:27.616871 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:27.616938 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:27.616967 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:27.617129 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:27.617440 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 16:00:27.617602 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":367,"threads_started":1}
I20260808 16:00:27.622529 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:27.624897 18213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9236068e4bc94d24b9047d1e40b21b52" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46845 } }
I20260808 16:00:27.625285 18213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:27.625437 18213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9236068e4bc94d24b9047d1e40b21b52, State: Initialized, Role: FOLLOWER
I20260808 16:00:27.626040 18213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9236068e4bc94d24b9047d1e40b21b52" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46845 } }
I20260808 16:00:27.626278 18213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:27.626426 18213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:27.626680 18213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:27.629803 18213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9236068e4bc94d24b9047d1e40b21b52" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46845 } }
I20260808 16:00:27.630380 18213 leader_election.cc:302] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9236068e4bc94d24b9047d1e40b21b52; no voters:
I20260808 16:00:27.630995 18213 leader_election.cc:288] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:27.631090 18223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:27.633458 18213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:27.633829 18223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [term 1 LEADER]: Becoming Leader. State: Replica: 9236068e4bc94d24b9047d1e40b21b52, State: Running, Role: LEADER
I20260808 16:00:27.634406 18223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9236068e4bc94d24b9047d1e40b21b52" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46845 } }
I20260808 16:00:27.634932 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:27.626855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36945 (local address 127.27.254.62:46845)
0808 16:00:27.627052 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:27.627061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:27.627520 (+ 459us) server_negotiation.cc:407] Connection header received
0808 16:00:27.627719 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:27.627726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:27.627786 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:27.627863 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:27.628449 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:27.629192 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:27.631234 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:27.632211 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:27.632369 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:27.633744 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:27.633751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:27.633756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:27.633777 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:27.633849 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:27.633872 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:27.634011 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:27.634485 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 16:00:27.634538 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 16:00:27.638305 18224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9236068e4bc94d24b9047d1e40b21b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9236068e4bc94d24b9047d1e40b21b52" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46845 } } }
I20260808 16:00:27.638671 18224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:27.640115 18225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9236068e4bc94d24b9047d1e40b21b52. Latest consensus state: current_term: 1 leader_uuid: "9236068e4bc94d24b9047d1e40b21b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9236068e4bc94d24b9047d1e40b21b52" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 46845 } } }
I20260808 16:00:27.640331 18225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:27.641636 18235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:27.645782 18235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:27.653741 18235 catalog_manager.cc:1347] Generated new cluster ID: b0e8c7567a6d4f4499430d787b311248
I20260808 16:00:27.653847 18235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:27.671185 18235 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:27.672967 18235 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:27.680557 18235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9236068e4bc94d24b9047d1e40b21b52: Generated new TSK 0
I20260808 16:00:27.681536 18235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:27.692669 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18145
W20260808 16:00:27.693773 18208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46657
W20260808 16:00:27.694456 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18145: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:27.694628 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18145
W20260808 16:00:27.710834 18130 connection.cc:535] client connection to 127.27.254.62:46845 recv error: Network error: failed to read from TLS socket (remote: 127.27.254.62:46845): Connection reset by peer (error 104)
2026-08-08T16:00:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:27 dist-test-slave-srgv krb5kdc[18244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:27 dist-test-slave-srgv krb5kdc[18244](info): set up 2 sockets
Aug 08 16:00:27 dist-test-slave-srgv krb5kdc[18244](info): commencing operation
krb5kdc: starting...
W20260808 16:00:28.787874 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:28 dist-test-slave-srgv krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:29.039922 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40163
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:40163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:29.356954 18257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:29.357275 18257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:29.357453 18257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:29.364440 18257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:29.364589 18257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:29.364703 18257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:29.364826 18257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:29.364907 18257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:29.374284 18257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:40163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:29.375505 18257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:29.376751 18257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:29.388053 18262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:29.388092 18265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:29.388144 18263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:29.388983 18257 server_base.cc:1034] running on GCE node
I20260808 16:00:29.389737 18257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:29.391057 18257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:29.392263 18257 hybrid_clock.cc:648] HybridClock initialized: now 1786204829392234 us; error 36 us; skew 500 ppm
I20260808 16:00:29.392539 18257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:29 dist-test-slave-srgv krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204829, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:29.400429 18257 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:29.401876 18257 webserver.cc:460] Webserver started at http://127.27.254.62:33583/ using document root <none> and password file <none>
I20260808 16:00:29.402473 18257 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:29.402588 18257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:29.402954 18257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:29.405900 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "934c97bef3a04854b60e5901f9ddea62"
format_stamp: "Formatted at 2026-08-08 16:00:29 on dist-test-slave-srgv"
server_key: "5d62737471991f8ba10219b2f9eaa27b"
server_key_iv: "3e558268622bf811dbab6469c3be4170"
server_key_version: "encryptionkey@0"
I20260808 16:00:29.406618 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "934c97bef3a04854b60e5901f9ddea62"
format_stamp: "Formatted at 2026-08-08 16:00:29 on dist-test-slave-srgv"
server_key: "5d62737471991f8ba10219b2f9eaa27b"
server_key_iv: "3e558268622bf811dbab6469c3be4170"
server_key_version: "encryptionkey@0"
I20260808 16:00:29.411432 18257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 16:00:29.414633 18272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:29.415920 18257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:29.416103 18257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "934c97bef3a04854b60e5901f9ddea62"
format_stamp: "Formatted at 2026-08-08 16:00:29 on dist-test-slave-srgv"
server_key: "5d62737471991f8ba10219b2f9eaa27b"
server_key_iv: "3e558268622bf811dbab6469c3be4170"
server_key_version: "encryptionkey@0"
I20260808 16:00:29.416283 18257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:29.421612 18257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:29.425158 18257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:29.425482 18257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:29.442802 18257 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:40163
I20260808 16:00:29.442870 18324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:40163 every 8 connection(s)
I20260808 16:00:29.443984 18257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:29.447309 18325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:29.453118 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18257
I20260808 16:00:29.453333 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:29.453778 28664 external_mini_cluster.cc:1428] Setting key 7748595e5bb335a18b283398d3c08851
I20260808 16:00:29.457154 18325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62: Bootstrap starting.
I20260808 16:00:29.460662 18325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:29.461318 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:29.455502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54503 (local address 127.27.254.62:40163)
0808 16:00:29.456290 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:29.456300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:29.456336 (+ 36us) server_negotiation.cc:407] Connection header received
0808 16:00:29.458053 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:29.458074 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:29.458371 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:29.458652 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:29.459755 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:29.459761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:29.459773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:29.459787 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:29.459841 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:29.459861 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:29.459993 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:29.460172 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 16:00:29.460364 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":357,"threads_started":1}
I20260808 16:00:29.461818 18325 log.cc:824] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:29.464835 18325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62: No bootstrap required, opened a new log
I20260808 16:00:29.465027 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:29.472847 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:29.470204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51859 (local address 127.27.254.62:40163)
0808 16:00:29.470428 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:29.470436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:29.470962 (+ 526us) server_negotiation.cc:407] Connection header received
0808 16:00:29.471151 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:29.471155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:29.471203 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:29.471319 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:29.472196 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:29.472201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:29.472206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:29.472227 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:29.472277 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:29.472292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:29.472437 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:29.472532 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 16:00:29.472588 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 16:00:29.474071 18325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934c97bef3a04854b60e5901f9ddea62" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40163 } }
I20260808 16:00:29.474515 18325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:29.474628 18325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934c97bef3a04854b60e5901f9ddea62, State: Initialized, Role: FOLLOWER
I20260808 16:00:29.475179 18325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934c97bef3a04854b60e5901f9ddea62" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40163 } }
I20260808 16:00:29.475417 18325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:29.475567 18325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:29.475750 18325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:29.478175 18325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934c97bef3a04854b60e5901f9ddea62" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40163 } }
I20260808 16:00:29.478706 18325 leader_election.cc:302] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 934c97bef3a04854b60e5901f9ddea62; no voters:
I20260808 16:00:29.479295 18325 leader_election.cc:288] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:29.479434 18335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:29.480123 18335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [term 1 LEADER]: Becoming Leader. State: Replica: 934c97bef3a04854b60e5901f9ddea62, State: Running, Role: LEADER
I20260808 16:00:29.480568 18335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934c97bef3a04854b60e5901f9ddea62" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40163 } }
I20260808 16:00:29.481276 18325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:29.483455 18336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "934c97bef3a04854b60e5901f9ddea62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934c97bef3a04854b60e5901f9ddea62" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40163 } } }
I20260808 16:00:29.483434 18337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 934c97bef3a04854b60e5901f9ddea62. Latest consensus state: current_term: 1 leader_uuid: "934c97bef3a04854b60e5901f9ddea62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934c97bef3a04854b60e5901f9ddea62" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 40163 } } }
I20260808 16:00:29.483958 18337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:29.483955 18336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:29.486205 18340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:29.491215 18340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:29.500839 18340 catalog_manager.cc:1347] Generated new cluster ID: d243e1e1ef1a4cd9b602d9e9e95a8ea3
I20260808 16:00:29.500954 18340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:29.517432 18340 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:29.517858 18340 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:29.526151 18340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 934c97bef3a04854b60e5901f9ddea62: Generated new TSK 0
I20260808 16:00:29.526960 18340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:29.531476 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18257
W20260808 16:00:29.532428 18320 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54503
W20260808 16:00:29.533108 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18257: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:29.533285 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18257
2026-08-08T16:00:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1838 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:29 dist-test-slave-srgv krb5kdc[18356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:29 dist-test-slave-srgv krb5kdc[18356](info): set up 2 sockets
Aug 08 16:00:29 dist-test-slave-srgv krb5kdc[18356](info): commencing operation
krb5kdc: starting...
W20260808 16:00:30.635602 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:30 dist-test-slave-srgv krb5kdc[18356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:30.894511 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33601
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:31.217972 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:31.218293 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:31.218461 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:31.225564 18369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:31.225720 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:31.225824 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:31.225936 18369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:31.226024 18369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:31.235431 18369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:31.236697 18369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:31.237989 18369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:31.249758 18377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:31.249758 18375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:31.250386 18369 server_base.cc:1034] running on GCE node
W20260808 16:00:31.250442 18374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:31.251245 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:31.252509 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:31.253731 18369 hybrid_clock.cc:648] HybridClock initialized: now 1786204831253681 us; error 42 us; skew 500 ppm
I20260808 16:00:31.254067 18369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:31 dist-test-slave-srgv krb5kdc[18356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:31.257246 18369 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:31.258677 18369 webserver.cc:460] Webserver started at http://127.27.254.62:43945/ using document root <none> and password file <none>
I20260808 16:00:31.259359 18369 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:31.259526 18369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:31.259847 18369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:31.262708 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "5b964361449b440daab310f51b0a8ae4"
format_stamp: "Formatted at 2026-08-08 16:00:31 on dist-test-slave-srgv"
server_key: "b1a933568528a22d732d01807687170b"
server_key_iv: "69c1fb94dc124bd4abcb4192dcc771fc"
server_key_version: "encryptionkey@0"
I20260808 16:00:31.263449 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "5b964361449b440daab310f51b0a8ae4"
format_stamp: "Formatted at 2026-08-08 16:00:31 on dist-test-slave-srgv"
server_key: "b1a933568528a22d732d01807687170b"
server_key_iv: "69c1fb94dc124bd4abcb4192dcc771fc"
server_key_version: "encryptionkey@0"
I20260808 16:00:31.268678 18369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 16:00:31.272022 18384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:31.273389 18369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 16:00:31.273626 18369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "5b964361449b440daab310f51b0a8ae4"
format_stamp: "Formatted at 2026-08-08 16:00:31 on dist-test-slave-srgv"
server_key: "b1a933568528a22d732d01807687170b"
server_key_iv: "69c1fb94dc124bd4abcb4192dcc771fc"
server_key_version: "encryptionkey@0"
I20260808 16:00:31.273820 18369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:31.297833 18369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:31.302192 18369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:31.302475 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:31.321125 18369 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33601
I20260808 16:00:31.321156 18435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33601 every 8 connection(s)
I20260808 16:00:31.322485 18369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:31.326014 18436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:31.330854 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18369
I20260808 16:00:31.331039 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:31.331545 28664 external_mini_cluster.cc:1428] Setting key 9b83197caf02880759072baa5cad3d21
I20260808 16:00:31.336853 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4: Bootstrap starting.
I20260808 16:00:31.340487 18436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:31.341746 18436 log.cc:824] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:31.345157 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4: No bootstrap required, opened a new log
Aug 08 16:00:31 dist-test-slave-srgv krb5kdc[18356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 16:00:31.355747 18436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b964361449b440daab310f51b0a8ae4" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33601 } }
I20260808 16:00:31.356189 18436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:31.356297 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:31.333576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57765 (local address 127.27.254.62:33601)
0808 16:00:31.334478 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:31.334492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:31.334531 (+ 39us) server_negotiation.cc:407] Connection header received
0808 16:00:31.336186 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:31.336208 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:31.336587 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:31.336900 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:31.338681 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:31.339967 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:31.341841 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:31.342907 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:31.343098 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:31.347727 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:31.347739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:31.347762 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:31.347817 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:31.351784 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:31.352515 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:31.352528 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:31.352537 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:31.352760 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:31.353108 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:31.353113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:31.353117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:31.353893 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:31.354199 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:31.354526 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 16:00:31.354976 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":441,"threads_started":1}
I20260808 16:00:31.356329 18436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b964361449b440daab310f51b0a8ae4, State: Initialized, Role: FOLLOWER
I20260808 16:00:31.357012 18436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b964361449b440daab310f51b0a8ae4" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33601 } }
I20260808 16:00:31.357277 18436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:31.357503 18436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:31.357687 18436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:31.360059 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:31.360240 18436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b964361449b440daab310f51b0a8ae4" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33601 } }
I20260808 16:00:31.360749 18436 leader_election.cc:302] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b964361449b440daab310f51b0a8ae4; no voters:
I20260808 16:00:31.361392 18436 leader_election.cc:288] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:31.361559 18441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:31.363543 18436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:31.363911 18441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [term 1 LEADER]: Becoming Leader. State: Replica: 5b964361449b440daab310f51b0a8ae4, State: Running, Role: LEADER
I20260808 16:00:31.364696 18441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b964361449b440daab310f51b0a8ae4" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33601 } }
I20260808 16:00:31.378310 18443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b964361449b440daab310f51b0a8ae4. Latest consensus state: current_term: 1 leader_uuid: "5b964361449b440daab310f51b0a8ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b964361449b440daab310f51b0a8ae4" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33601 } } }
I20260808 16:00:31.379881 18443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:31.379737 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b964361449b440daab310f51b0a8ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b964361449b440daab310f51b0a8ae4" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33601 } } }
I20260808 16:00:31.380597 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:31.381893 18458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:31.387390 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:31.366169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36035 (local address 127.27.254.62:33601)
0808 16:00:31.366352 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:31.366358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:31.367038 (+ 680us) server_negotiation.cc:407] Connection header received
0808 16:00:31.367279 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:31.367288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:31.367389 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:31.367520 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:31.369895 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:31.371350 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:31.373557 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:31.374186 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:31.374333 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:31.378923 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:31.378934 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:31.378941 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:31.378977 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:31.383629 (+ 4652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:31.384258 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:31.384263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:31.384267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:31.384429 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:31.384990 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:31.384996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:31.385002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:31.385373 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:31.386637 (+ 1264us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:31.386798 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 16:00:31.387021 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 16:00:31.388782 18458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:31.398470 18458 catalog_manager.cc:1347] Generated new cluster ID: 8e39720dc52c40eda217de7add40d506
I20260808 16:00:31.398584 18458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:31.419735 18458 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:31.421689 18458 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:31.429368 18458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b964361449b440daab310f51b0a8ae4: Generated new TSK 0
I20260808 16:00:31.430398 18458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:31.440639 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18369
I20260808 16:00:31.441762 18431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:31.532087 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18369
2026-08-08T16:00:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:31 dist-test-slave-srgv krb5kdc[18470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:31 dist-test-slave-srgv krb5kdc[18470](info): set up 2 sockets
Aug 08 16:00:31 dist-test-slave-srgv krb5kdc[18470](info): commencing operation
krb5kdc: starting...
W20260808 16:00:32.623594 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:32 dist-test-slave-srgv krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:32.885244 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41107
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:41107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:33.209555 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:33.209867 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:33.210023 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:33.217054 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:33.217152 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:33.217245 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:33.217352 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:33.217444 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:33.227090 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:41107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:41107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:33.228351 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:33.229720 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:33.241187 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:33.241241 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:33.241160 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:33.242477 18483 server_base.cc:1034] running on GCE node
I20260808 16:00:33.243160 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:33.244474 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:33.245663 18483 hybrid_clock.cc:648] HybridClock initialized: now 1786204833245628 us; error 29 us; skew 500 ppm
I20260808 16:00:33.245957 18483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:33 dist-test-slave-srgv krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204833, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:33.248921 18483 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:33.250349 18483 webserver.cc:460] Webserver started at http://127.27.254.62:43939/ using document root <none> and password file <none>
I20260808 16:00:33.250998 18483 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:33.251093 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:33.251411 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:33.254390 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "fe641f069ede4a23b147e5f0ac377e4d"
format_stamp: "Formatted at 2026-08-08 16:00:33 on dist-test-slave-srgv"
server_key: "ef801956e75315f08bf4475726a3b848"
server_key_iv: "0b7ca1c8b44b1cbbed15aec9892d43ee"
server_key_version: "encryptionkey@0"
I20260808 16:00:33.255117 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "fe641f069ede4a23b147e5f0ac377e4d"
format_stamp: "Formatted at 2026-08-08 16:00:33 on dist-test-slave-srgv"
server_key: "ef801956e75315f08bf4475726a3b848"
server_key_iv: "0b7ca1c8b44b1cbbed15aec9892d43ee"
server_key_version: "encryptionkey@0"
I20260808 16:00:33.260267 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 16:00:33.263521 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:33.264783 18483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 16:00:33.264921 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "fe641f069ede4a23b147e5f0ac377e4d"
format_stamp: "Formatted at 2026-08-08 16:00:33 on dist-test-slave-srgv"
server_key: "ef801956e75315f08bf4475726a3b848"
server_key_iv: "0b7ca1c8b44b1cbbed15aec9892d43ee"
server_key_version: "encryptionkey@0"
I20260808 16:00:33.265096 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:33.294131 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:33.298221 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:33.298456 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:33.316329 18483 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:41107
I20260808 16:00:33.316363 18549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:41107 every 8 connection(s)
I20260808 16:00:33.317757 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:33.321319 18550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:33.322662 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18483
I20260808 16:00:33.322865 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:33.323320 28664 external_mini_cluster.cc:1428] Setting key c5aa337ccd793fdaa1de6d7d0c899262
I20260808 16:00:33.335352 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d: Bootstrap starting.
I20260808 16:00:33.338580 18550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:33.339895 18550 log.cc:824] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:33.341029 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:33.325001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45807 (local address 127.27.254.62:41107)
0808 16:00:33.325814 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:33.325831 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:33.325880 (+ 49us) server_negotiation.cc:407] Connection header received
0808 16:00:33.327711 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:33.327738 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:33.328266 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:33.328687 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:33.329160 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:33.331063 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:33.332347 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:33.333494 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:33.333664 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:33.338299 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:33.338310 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:33.338329 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:33.339090 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:33.339199 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:33.339229 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:33.339338 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:33.339561 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 16:00:33.339845 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":350,"threads_started":1}
I20260808 16:00:33.342937 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d: No bootstrap required, opened a new log
I20260808 16:00:33.344929 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:33.353133 18550 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe641f069ede4a23b147e5f0ac377e4d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41107 } }
I20260808 16:00:33.353560 18550 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:33.353653 18550 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe641f069ede4a23b147e5f0ac377e4d, State: Initialized, Role: FOLLOWER
I20260808 16:00:33.354311 18550 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe641f069ede4a23b147e5f0ac377e4d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41107 } }
I20260808 16:00:33.354604 18550 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:33.354771 18550 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:33.354979 18550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:33.356814 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:33.350018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44579 (local address 127.27.254.62:41107)
0808 16:00:33.350211 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:33.350217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:33.350797 (+ 580us) server_negotiation.cc:407] Connection header received
0808 16:00:33.350922 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:33.350927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:33.351006 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:33.351114 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:33.351580 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:33.352668 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:33.354189 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:33.354901 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:33.355049 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:33.356096 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:33.356102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:33.356107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:33.356129 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:33.356177 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:33.356197 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:33.356313 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:33.356416 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 16:00:33.356503 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 16:00:33.358134 18550 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe641f069ede4a23b147e5f0ac377e4d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41107 } }
I20260808 16:00:33.358621 18550 leader_election.cc:302] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe641f069ede4a23b147e5f0ac377e4d; no voters:
I20260808 16:00:33.359191 18550 leader_election.cc:288] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:33.359323 18560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:33.360009 18560 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [term 1 LEADER]: Becoming Leader. State: Replica: fe641f069ede4a23b147e5f0ac377e4d, State: Running, Role: LEADER
I20260808 16:00:33.360500 18560 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe641f069ede4a23b147e5f0ac377e4d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41107 } }
I20260808 16:00:33.361490 18550 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:33.363438 18562 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe641f069ede4a23b147e5f0ac377e4d. Latest consensus state: current_term: 1 leader_uuid: "fe641f069ede4a23b147e5f0ac377e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe641f069ede4a23b147e5f0ac377e4d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41107 } } }
I20260808 16:00:33.363776 18562 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:33.364658 18566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:33.369179 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe641f069ede4a23b147e5f0ac377e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe641f069ede4a23b147e5f0ac377e4d" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 41107 } } }
I20260808 16:00:33.369969 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:33.374907 18566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:33.383764 18566 catalog_manager.cc:1347] Generated new cluster ID: 527c4c7918054a84859b68a2aef339ac
I20260808 16:00:33.383878 18566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:33.437096 18566 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:33.438460 18566 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:33.453918 18566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe641f069ede4a23b147e5f0ac377e4d: Generated new TSK 0
I20260808 16:00:33.454978 18566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:33.473604 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18483
W20260808 16:00:33.474835 18545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45807
W20260808 16:00:33.475660 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:33.475800 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18483
2026-08-08T16:00:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:33 dist-test-slave-srgv krb5kdc[18581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:33 dist-test-slave-srgv krb5kdc[18581](info): set up 2 sockets
Aug 08 16:00:33 dist-test-slave-srgv krb5kdc[18581](info): commencing operation
krb5kdc: starting...
W20260808 16:00:34.603897 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:34 dist-test-slave-srgv krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:34.860340 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36207
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:36207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:35.189203 18594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:35.189491 18594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:35.189661 18594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:35.196805 18594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:35.196907 18594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:35.196952 18594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:35.197031 18594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:35.197096 18594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:35.206924 18594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:36207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:36207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:35.208158 18594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:35.209452 18594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:35.221344 18599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:35.221463 18600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:35.221354 18602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:35.222267 18594 server_base.cc:1034] running on GCE node
I20260808 16:00:35.223043 18594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:35.224391 18594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:35.225616 18594 hybrid_clock.cc:648] HybridClock initialized: now 1786204835225581 us; error 47 us; skew 500 ppm
I20260808 16:00:35.225967 18594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:35 dist-test-slave-srgv krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204835, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:35.233606 18594 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:35.235150 18594 webserver.cc:460] Webserver started at http://127.27.254.62:45011/ using document root <none> and password file <none>
I20260808 16:00:35.235832 18594 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:35.235980 18594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:35.236287 18594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:35.239231 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "8a7a73272c744496a7cf15a4c745c9dc"
format_stamp: "Formatted at 2026-08-08 16:00:35 on dist-test-slave-srgv"
server_key: "28fbb94a2627301d0731d582d31ee33a"
server_key_iv: "4d7a1e826e0e1460fdfd69c52427cfab"
server_key_version: "encryptionkey@0"
I20260808 16:00:35.239980 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "8a7a73272c744496a7cf15a4c745c9dc"
format_stamp: "Formatted at 2026-08-08 16:00:35 on dist-test-slave-srgv"
server_key: "28fbb94a2627301d0731d582d31ee33a"
server_key_iv: "4d7a1e826e0e1460fdfd69c52427cfab"
server_key_version: "encryptionkey@0"
I20260808 16:00:35.245023 18594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 16:00:35.248376 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:35.249737 18594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:35.249914 18594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "8a7a73272c744496a7cf15a4c745c9dc"
format_stamp: "Formatted at 2026-08-08 16:00:35 on dist-test-slave-srgv"
server_key: "28fbb94a2627301d0731d582d31ee33a"
server_key_iv: "4d7a1e826e0e1460fdfd69c52427cfab"
server_key_version: "encryptionkey@0"
I20260808 16:00:35.250114 18594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:35.255631 18594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:35.259446 18594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:35.259739 18594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:35.277920 18594 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:36207
I20260808 16:00:35.277957 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:36207 every 8 connection(s)
I20260808 16:00:35.279310 18594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:35.282996 18663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:35.285022 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18594
I20260808 16:00:35.285210 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:35.285799 28664 external_mini_cluster.cc:1428] Setting key 02d193600c0d1a372d1bffa8f934c910
I20260808 16:00:35.295578 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:35.287869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55407 (local address 127.27.254.62:36207)
0808 16:00:35.288855 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:35.288875 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:35.289313 (+ 438us) server_negotiation.cc:407] Connection header received
0808 16:00:35.291741 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:35.291769 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:35.292212 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:35.292670 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:35.293673 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:35.293681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:35.293697 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:35.293719 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:35.293796 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:35.293826 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:35.294042 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:35.294183 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 16:00:35.294399 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":424,"threads_started":1}
I20260808 16:00:35.296566 18663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc: Bootstrap starting.
I20260808 16:00:35.299299 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:35.299898 18663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:35.301106 18663 log.cc:824] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:35.304320 18663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc: No bootstrap required, opened a new log
I20260808 16:00:35.306813 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:35.304568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53373 (local address 127.27.254.62:36207)
0808 16:00:35.304744 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:35.304748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:35.304877 (+ 129us) server_negotiation.cc:407] Connection header received
0808 16:00:35.305048 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:35.305051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:35.305092 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:35.305168 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:35.306134 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:35.306138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:35.306141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 16:00:35.306154 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:35.306196 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 16:00:35.306210 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:35.306314 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:35.306440 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 16:00:35.306494 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 16:00:35.312325 18663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a73272c744496a7cf15a4c745c9dc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36207 } }
I20260808 16:00:35.312690 18663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 16:00:35.312798 18663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7a73272c744496a7cf15a4c745c9dc, State: Initialized, Role: FOLLOWER
I20260808 16:00:35.313347 18663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a73272c744496a7cf15a4c745c9dc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36207 } }
I20260808 16:00:35.313638 18663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 16:00:35.313786 18663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 16:00:35.313959 18663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:35.316064 18663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a73272c744496a7cf15a4c745c9dc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36207 } }
I20260808 16:00:35.316534 18663 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7a73272c744496a7cf15a4c745c9dc; no voters:
I20260808 16:00:35.317090 18663 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 16:00:35.317271 18673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:35.317942 18673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [term 1 LEADER]: Becoming Leader. State: Replica: 8a7a73272c744496a7cf15a4c745c9dc, State: Running, Role: LEADER
I20260808 16:00:35.318500 18673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a73272c744496a7cf15a4c745c9dc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36207 } }
I20260808 16:00:35.319306 18663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:35.321302 18675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7a73272c744496a7cf15a4c745c9dc. Latest consensus state: current_term: 1 leader_uuid: "8a7a73272c744496a7cf15a4c745c9dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a73272c744496a7cf15a4c745c9dc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36207 } } }
I20260808 16:00:35.321833 18675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:35.321305 18674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7a73272c744496a7cf15a4c745c9dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a73272c744496a7cf15a4c745c9dc" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 36207 } } }
I20260808 16:00:35.322292 18674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:35.333691 18678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:35.337698 18678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:35.345486 18678 catalog_manager.cc:1347] Generated new cluster ID: 63b20a2ca7e940959d7c78a0d52ec56e
I20260808 16:00:35.345597 18678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:35.359601 18678 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:35.360095 18678 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:35.373073 18678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7a73272c744496a7cf15a4c745c9dc: Generated new TSK 0
I20260808 16:00:35.374049 18678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 16:00:35.387786 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18594
W20260808 16:00:35.388975 18658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55407
W20260808 16:00:35.389691 28664 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18594: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 16:00:35.389842 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18594
2026-08-08T16:00:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1906 ms)
[----------] 18 tests from AuthTokenIssuingTest (34402 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:35 dist-test-slave-srgv krb5kdc[18694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:35 dist-test-slave-srgv krb5kdc[18694](info): set up 2 sockets
Aug 08 16:00:35 dist-test-slave-srgv krb5kdc[18694](info): commencing operation
krb5kdc: starting...
W20260808 16:00:36.491899 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:36 dist-test-slave-srgv krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:36.756810 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33597
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33597,127.27.254.61:38661,127.27.254.60:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:37.084134 18707 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 16:00:37.084518 18707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:37.084640 18707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:37.084782 18707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:37.092175 18707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:37.092289 18707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:37.092411 18707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:37.092495 18707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:37.092583 18707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:37.102367 18707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33597,127.27.254.61:38661,127.27.254.60:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:37.103659 18707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:37.105016 18707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:37.116819 18712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:37.116771 18713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:37.116845 18715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:37.117799 18707 server_base.cc:1034] running on GCE node
I20260808 16:00:37.118506 18707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:37.119832 18707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:37.121078 18707 hybrid_clock.cc:648] HybridClock initialized: now 1786204837121001 us; error 56 us; skew 500 ppm
I20260808 16:00:37.121397 18707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:37 dist-test-slave-srgv krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:37.124433 18707 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:37.125835 18707 webserver.cc:460] Webserver started at http://127.27.254.62:34273/ using document root <none> and password file <none>
I20260808 16:00:37.126436 18707 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:37.126519 18707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:37.126782 18707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:37.129563 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "3be044fc9b74461496ed466ac30ca80f"
format_stamp: "Formatted at 2026-08-08 16:00:37 on dist-test-slave-srgv"
server_key: "e5c4abd7285971fe80264fa1cf985891"
server_key_iv: "2b77b89062fbe5094f4c4b31448a5528"
server_key_version: "encryptionkey@0"
I20260808 16:00:37.130293 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "3be044fc9b74461496ed466ac30ca80f"
format_stamp: "Formatted at 2026-08-08 16:00:37 on dist-test-slave-srgv"
server_key: "e5c4abd7285971fe80264fa1cf985891"
server_key_iv: "2b77b89062fbe5094f4c4b31448a5528"
server_key_version: "encryptionkey@0"
I20260808 16:00:37.135221 18707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 16:00:37.138427 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:37.139684 18707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 16:00:37.139880 18707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "3be044fc9b74461496ed466ac30ca80f"
format_stamp: "Formatted at 2026-08-08 16:00:37 on dist-test-slave-srgv"
server_key: "e5c4abd7285971fe80264fa1cf985891"
server_key_iv: "2b77b89062fbe5094f4c4b31448a5528"
server_key_version: "encryptionkey@0"
I20260808 16:00:37.140081 18707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:37.156250 18707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:37.159973 18707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:37.160234 18707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:37.178320 18707 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:33597
I20260808 16:00:37.178344 18774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:33597 every 8 connection(s)
I20260808 16:00:37.179582 18707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:37.181483 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18707
I20260808 16:00:37.181718 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:37.182543 28664 external_mini_cluster.cc:1428] Setting key cfee81fd02735bd4aa0c658be5b272bb
I20260808 16:00:37.183954 18775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:37.193665 18775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:37 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.61@KRBTEST.COM" created.
I20260808 16:00:37.216617 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.196977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.27.254.62:33597)
0808 16:00:37.197872 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:37.197882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:37.198423 (+ 541us) server_negotiation.cc:407] Connection header received
0808 16:00:37.200380 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:37.200386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:37.200606 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:37.200812 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:37.202122 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.203253 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.204630 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.205195 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.205432 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.209193 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:37.209208 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:37.209219 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:37.209247 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:37.212100 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.212891 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.212906 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.212908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.213050 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.213518 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.213527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.213529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.214003 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:37.214214 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:37.214756 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 16:00:37.215158 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":702,"thread_start_us":481,"threads_started":1}
I20260808 16:00:37.216635 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.196984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33597 (local address 127.0.0.1:59370)
0808 16:00:37.198268 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.198307 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:37.200005 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:37.200835 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:37.200841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:37.201603 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:37.201922 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.201938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.203408 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.203417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.204492 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.204503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.205456 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.205463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.205645 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.206313 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:37.206334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:37.208974 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:37.212230 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.212240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.212249 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.212760 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.213164 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.213171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.213175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.213366 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.214158 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:37.214165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:37.214390 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 16:00:37.214627 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 16:00:37.215277 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":188,"thread_start_us":932,"threads_started":1}
I20260808 16:00:37.219642 18775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:37.220716 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.220327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34580)
0808 16:00:37.220498 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.220610 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 16:00:37.220986 18725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
I20260808 16:00:37.222452 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.222138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34584)
0808 16:00:37.222275 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.222337 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 16:00:37.222751 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:38661: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab.
Entry for principal kudu/127.27.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab.
I20260808 16:00:37.242784 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:38661
--webserver_interface=127.27.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab
--principal=kudu/127.27.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33597,127.27.254.61:38661,127.27.254.60:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 16:00:37.272122 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } attempt: 1
I20260808 16:00:37.273308 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.272860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34592)
0808 16:00:37.273053 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.273176 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 16:00:37.274354 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.273995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34596)
0808 16:00:37.274161 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.274229 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 16:00:37.274562 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:38661: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
I20260808 16:00:37.342805 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } attempt: 2
I20260808 16:00:37.343997 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.343491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34606)
0808 16:00:37.343741 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.343833 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 16:00:37.345041 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.344704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34622)
0808 16:00:37.344854 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.344929 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 16:00:37.345217 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:38661: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
I20260808 16:00:37.436450 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } attempt: 3
I20260808 16:00:37.437718 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.437110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34626)
0808 16:00:37.437298 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.437579 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 16:00:37.438761 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.438394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34636)
0808 16:00:37.438563 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.438630 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 16:00:37.438928 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:38661: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
W20260808 16:00:37.562186 18782 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 16:00:37.562541 18782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:37.562667 18782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:37.562806 18782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:37.569507 18782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:37.569653 18782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:37.569762 18782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:37.569850 18782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:37.569939 18782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:37.578990 18782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33597,127.27.254.61:38661,127.27.254.60:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab
--principal=kudu/127.27.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:37.580199 18782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:37.581490 18782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 16:00:37.582141 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } attempt: 4
I20260808 16:00:37.583352 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.582820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34638)
0808 16:00:37.583138 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.583226 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260808 16:00:37.584318 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.583975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34646)
0808 16:00:37.584120 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.584201 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 16:00:37.584501 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:38661: Network error: Client connection negotiation failed: client connection to 127.27.254.61:38661: connect: Connection refused (error 111)
W20260808 16:00:37.593178 18787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:37.593219 18790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:37.593148 18788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:37.594272 18782 server_base.cc:1034] running on GCE node
I20260808 16:00:37.595042 18782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:37.596271 18782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:37.597509 18782 hybrid_clock.cc:648] HybridClock initialized: now 1786204837597451 us; error 25 us; skew 500 ppm
I20260808 16:00:37.597779 18782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:37 dist-test-slave-srgv krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:37.600813 18782 init.cc:377] Logged in from keytab as kudu/127.27.254.61@KRBTEST.COM (short username kudu)
I20260808 16:00:37.602226 18782 webserver.cc:460] Webserver started at http://127.27.254.61:44093/ using document root <none> and password file <none>
I20260808 16:00:37.602878 18782 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:37.603000 18782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:37.603314 18782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:37.606148 18782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data/instance:
uuid: "e35e30ed37e140568c83ebc1105fdc90"
format_stamp: "Formatted at 2026-08-08 16:00:37 on dist-test-slave-srgv"
server_key: "94707e9f825563ec39b3e0780ed6d738"
server_key_iv: "d15cd1335beaba72cc92a6113b7ed5d7"
server_key_version: "encryptionkey@0"
I20260808 16:00:37.606930 18782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal/instance:
uuid: "e35e30ed37e140568c83ebc1105fdc90"
format_stamp: "Formatted at 2026-08-08 16:00:37 on dist-test-slave-srgv"
server_key: "94707e9f825563ec39b3e0780ed6d738"
server_key_iv: "d15cd1335beaba72cc92a6113b7ed5d7"
server_key_version: "encryptionkey@0"
I20260808 16:00:37.611773 18782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 16:00:37.614851 18797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:37.616060 18782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 16:00:37.616236 18782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal
uuid: "e35e30ed37e140568c83ebc1105fdc90"
format_stamp: "Formatted at 2026-08-08 16:00:37 on dist-test-slave-srgv"
server_key: "94707e9f825563ec39b3e0780ed6d738"
server_key_iv: "d15cd1335beaba72cc92a6113b7ed5d7"
server_key_version: "encryptionkey@0"
I20260808 16:00:37.616442 18782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:37.642387 18782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:37.646289 18782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:37.646569 18782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:37.664758 18782 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.61:38661
I20260808 16:00:37.664778 18849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.61:38661 every 8 connection(s)
I20260808 16:00:37.666148 18782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
I20260808 16:00:37.666636 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18782
I20260808 16:00:37.666832 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-1/wal/instance
I20260808 16:00:37.667447 28664 external_mini_cluster.cc:1428] Setting key be5a54b5a87f49c61399ca5224fcfd12
I20260808 16:00:37.669938 18850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:37.676036 18850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:37 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.60@KRBTEST.COM" created.
I20260808 16:00:37.699452 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.678537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.27.254.62:33597)
0808 16:00:37.678753 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:37.678760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:37.679999 (+ 1239us) server_negotiation.cc:407] Connection header received
0808 16:00:37.681870 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:37.681879 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:37.681972 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:37.682096 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:37.683507 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.684661 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.686249 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.687053 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.687201 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.690780 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:37.690787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:37.690792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:37.690837 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:37.693708 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.694428 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.694436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.694443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.694588 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.697825 (+ 3237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.697829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.697831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.698097 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:37.698212 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:37.699068 (+ 856us) server_negotiation.cc:299] Negotiation successful
0808 16:00:37.699231 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 16:00:37.700987 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.679053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33597 (local address 127.0.0.1:59384)
0808 16:00:37.679810 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.679858 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:37.681612 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:37.682275 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:37.682283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:37.682767 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:37.683329 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.683345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.684828 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.684832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.686079 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.686091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.687256 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.687261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.687454 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.688154 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:37.688176 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:37.690540 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:37.693837 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.693843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.693854 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.694300 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.697469 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.697477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.697482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.697707 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.698225 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:37.698235 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:37.698687 (+ 452us) client_negotiation.cc:769] Sending connection context
0808 16:00:37.698978 (+ 291us) client_negotiation.cc:240] Negotiation successful
0808 16:00:37.699530 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":330,"threads_started":1}
I20260808 16:00:37.702781 18850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:37 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 16:00:37.718046 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.703710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34652)
0808 16:00:37.703917 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.703935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:37.704032 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:37.704906 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:37.704910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:37.705324 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:37.705550 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.705560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.707039 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.707046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.707869 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.707879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.708658 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.708664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.708846 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.709465 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:37.709482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:37.711414 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:37.715679 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.715683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.715686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.716034 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.716437 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.716440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.716443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.716573 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.717299 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:37.717302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:37.717499 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 16:00:37.717612 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 16:00:37.717803 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 16:00:37.718571 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.703934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.27.254.61:38661)
0808 16:00:37.704524 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:37.704530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:37.704552 (+ 22us) server_negotiation.cc:407] Connection header received
0808 16:00:37.704594 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:37.704599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:37.704765 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:37.704920 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:37.705729 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.706885 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.708004 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.708524 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.708653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.711666 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:37.711681 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:37.711699 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:37.711740 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:37.715545 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.716168 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.716179 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.716186 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.716330 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.716683 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.716692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.716698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.717176 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:37.717398 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:37.717804 (+ 406us) server_negotiation.cc:299] Negotiation successful
0808 16:00:37.718100 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":325,"threads_started":1}
I20260808 16:00:37.720991 18850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:37.722167 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.721755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45284)
0808 16:00:37.721909 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.722033 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 16:00:37.722451 18798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:37.724205 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.723825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45294)
0808 16:00:37.723993 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.724070 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 16:00:37.724412 18850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab.
Entry for principal kudu/127.27.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab.
I20260808 16:00:37.728780 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:35631
--webserver_interface=127.27.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab
--principal=kudu/127.27.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:33597,127.27.254.61:38661,127.27.254.60:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 16:00:37.773773 18850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 1
I20260808 16:00:37.775025 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.774552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45308)
0808 16:00:37.774756 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.774878 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 16:00:37.776049 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.775718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45322)
0808 16:00:37.775879 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.775948 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 16:00:37.776294 18850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:37.844542 18850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 2
I20260808 16:00:37.845716 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.845225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45328)
0808 16:00:37.845456 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.845553 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 16:00:37.846625 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.846336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45340)
0808 16:00:37.846463 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.846526 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 16:00:37.847018 18850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:37.883819 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } attempt: 5
Aug 08 16:00:37 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 16:00:37.897073 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.884702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.27.254.61:38661)
0808 16:00:37.884892 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:37.884898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:37.884994 (+ 96us) server_negotiation.cc:407] Connection header received
0808 16:00:37.885199 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:37.885206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:37.885289 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:37.885397 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:37.886221 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.886921 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.888088 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.888502 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.888637 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.892002 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:37.892007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:37.892011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:37.892040 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:37.894564 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.895308 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.895311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.895313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.895425 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:37.895881 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:37.895885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:37.895887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:37.896150 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:37.896259 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:37.896685 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 16:00:37.896850 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 16:00:37.897308 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.884652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34656)
0808 16:00:37.884874 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.884895 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:37.885067 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:37.885476 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:37.885483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:37.885888 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:37.886058 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.886067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.887056 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.887062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.887932 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:37.887943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:37.888651 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:37.888656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:37.888893 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:37.889550 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:37.889585 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:37.891810 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:37.894715 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.894720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.894724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.895206 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.895566 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:37.895572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:37.895577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:37.895764 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:37.896267 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:37.896272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:37.896438 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 16:00:37.896599 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 16:00:37.896961 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 16:00:37.898306 18775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:37.899334 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.898983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45348)
0808 16:00:37.899136 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.899210 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 16:00:37.900286 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.900007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45364)
0808 16:00:37.900122 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.900203 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 16:00:37.900573 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:37.938316 18850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 3
I20260808 16:00:37.939504 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.939039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45380)
0808 16:00:37.939262 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.939352 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 16:00:37.940544 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.940141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45388)
0808 16:00:37.940360 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.940436 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260808 16:00:37.940733 18850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:37.951808 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 1
I20260808 16:00:37.952939 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.952506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45398)
0808 16:00:37.952684 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.952776 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 16:00:37.953989 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:37.953660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45400)
0808 16:00:37.953812 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:37.953889 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 16:00:37.954207 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:38.022433 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 2
I20260808 16:00:38.023582 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.023080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45408)
0808 16:00:38.023321 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.023405 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 16:00:38.024713 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.024252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45424)
0808 16:00:38.024476 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.024573 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 16:00:38.024964 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
W20260808 16:00:38.064198 18857 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 16:00:38.064525 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:38.064636 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:38.064777 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:38.072109 18857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:38.072197 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:38.072309 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:38.072399 18857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:38.072523 18857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:38.082530 18857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:33597,127.27.254.61:38661,127.27.254.60:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab
--principal=kudu/127.27.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:38.083811 18857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:38.083964 18850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 4
I20260808 16:00:38.085064 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.084564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45440)
0808 16:00:38.084825 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.084908 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 16:00:38.085253 18857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 16:00:38.086056 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.085744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45456)
0808 16:00:38.085887 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.085961 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 16:00:38.086261 18850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
W20260808 16:00:38.097209 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:38.097185 18863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:38.097438 18865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:38.098130 18857 server_base.cc:1034] running on GCE node
I20260808 16:00:38.098860 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:38.100144 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:38.101397 18857 hybrid_clock.cc:648] HybridClock initialized: now 1786204838101315 us; error 46 us; skew 500 ppm
I20260808 16:00:38.101763 18857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:38.104888 18857 init.cc:377] Logged in from keytab as kudu/127.27.254.60@KRBTEST.COM (short username kudu)
I20260808 16:00:38.106300 18857 webserver.cc:460] Webserver started at http://127.27.254.60:41323/ using document root <none> and password file <none>
I20260808 16:00:38.106956 18857 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:38.107084 18857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:38.107452 18857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:38.110419 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data/instance:
uuid: "1b137e83445b4cc1804e69437943dc07"
format_stamp: "Formatted at 2026-08-08 16:00:38 on dist-test-slave-srgv"
server_key: "f749d8e530ee207396185ffd4b8b362a"
server_key_iv: "06a2cb8143838ecd980556346b9375c1"
server_key_version: "encryptionkey@0"
I20260808 16:00:38.111161 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal/instance:
uuid: "1b137e83445b4cc1804e69437943dc07"
format_stamp: "Formatted at 2026-08-08 16:00:38 on dist-test-slave-srgv"
server_key: "f749d8e530ee207396185ffd4b8b362a"
server_key_iv: "06a2cb8143838ecd980556346b9375c1"
server_key_version: "encryptionkey@0"
I20260808 16:00:38.115839 18857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 16:00:38.119098 18872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:38.120456 18857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 16:00:38.120626 18857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal
uuid: "1b137e83445b4cc1804e69437943dc07"
format_stamp: "Formatted at 2026-08-08 16:00:38 on dist-test-slave-srgv"
server_key: "f749d8e530ee207396185ffd4b8b362a"
server_key_iv: "06a2cb8143838ecd980556346b9375c1"
server_key_version: "encryptionkey@0"
I20260808 16:00:38.120841 18857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:38.131187 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 3
I20260808 16:00:38.132261 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.131772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45466)
0808 16:00:38.131974 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.132077 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 16:00:38.133263 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.132875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45472)
0808 16:00:38.133068 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.133144 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260808 16:00:38.133476 18775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:35631: Network error: Client connection negotiation failed: client connection to 127.27.254.60:35631: connect: Connection refused (error 111)
I20260808 16:00:38.137112 18857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:38.141188 18857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:38.141400 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:38.160037 18857 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.60:35631
I20260808 16:00:38.160061 18924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.60:35631 every 8 connection(s)
I20260808 16:00:38.161360 18857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
I20260808 16:00:38.164995 18925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:38.166998 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 18857
I20260808 16:00:38.167174 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204485174398-28664-0/minicluster-data/master-2/wal/instance
I20260808 16:00:38.167714 28664 external_mini_cluster.cc:1428] Setting key dd63f2cf1ac40a59bc3275d761a11c00
I20260808 16:00:38.174546 18925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 16:00:38.192569 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.169601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.27.254.60:35631)
0808 16:00:38.170423 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.170436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.170474 (+ 38us) server_negotiation.cc:407] Connection header received
0808 16:00:38.172023 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.172054 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.172424 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.172723 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.174556 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.175884 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.177705 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.178736 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.178956 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.183957 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.183972 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.183986 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.184029 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.187942 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.188811 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.188820 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.188828 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.189022 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.189469 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.189477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.189483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.190204 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.190513 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.190891 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.191275 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":401,"threads_started":1}
I20260808 16:00:38.193637 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.177224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.27.254.62:33597)
0808 16:00:38.177385 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.177389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.178095 (+ 706us) server_negotiation.cc:407] Connection header received
0808 16:00:38.178370 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.178376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.178459 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.178571 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.180049 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.181202 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.183015 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.183776 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.183921 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.187441 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.187449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.187457 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.187487 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.190987 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.191642 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.191648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.191651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.191809 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.192142 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.192147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.192150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.192494 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.192661 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.193170 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.193330 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 16:00:38.193774 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.177126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:33597 (local address 127.0.0.1:59394)
0808 16:00:38.177952 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.177999 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:38.178218 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:38.178634 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:38.178640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:38.179510 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:38.179873 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.179887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.181380 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.181385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.182843 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.182856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.183987 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.183997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.184189 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.184845 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:38.184883 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:38.187250 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:38.191128 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.191136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.191141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.191510 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.191915 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.191918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.191920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.192033 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.192620 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:38.192624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:38.192867 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 16:00:38.193059 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 16:00:38.193361 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":358,"threads_started":1}
I20260808 16:00:38.195096 18925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 16:00:38.209017 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.195845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:38661 (local address 127.0.0.1:34658)
0808 16:00:38.196079 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.196095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:38.196191 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:38.196540 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:38.196547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:38.197005 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:38.197188 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.197202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.198409 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.198412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.199570 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.199584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.200347 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.200351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.200489 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.201041 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:38.201057 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:38.203288 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:38.206558 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.206562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.206565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.207093 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.207460 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.207464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.207468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.207666 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.208192 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:38.208209 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:38.208366 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 16:00:38.208488 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 16:00:38.208689 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 16:00:38.209157 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.195850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.27.254.61:38661)
0808 16:00:38.195994 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.195998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.196141 (+ 143us) server_negotiation.cc:407] Connection header received
0808 16:00:38.196326 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.196331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.196410 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.196504 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.197345 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.198266 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.199738 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.200137 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.200265 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.203480 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.203485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.203489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.203520 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.206402 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.207210 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.207215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.207217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.207344 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.207809 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.207813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.207815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.208064 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.208180 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.208630 (+ 450us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.208864 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 16:00:38.210294 18925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:38.211072 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.195782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.27.254.62:33597)
0808 16:00:38.196056 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.196060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.196092 (+ 32us) server_negotiation.cc:407] Connection header received
0808 16:00:38.196230 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.196233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.196320 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.196414 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.197912 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.198806 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.200283 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.200696 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.200788 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.205003 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.205007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.205010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.205039 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.208107 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.208878 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.208883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.208887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.209032 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.209570 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.209576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.209582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.209949 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.210263 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.210551 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.210786 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 16:00:38.223449 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.211080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45480)
0808 16:00:38.211352 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.211367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:38.211443 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:38.211721 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:38.211724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:38.212037 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:38.212164 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.212171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.213201 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.213209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.214085 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.214095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.214763 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.214768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.214965 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.215524 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:38.215539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:38.217341 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:38.220971 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.220976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.220980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.221506 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.221863 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.221868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.221872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.222032 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.222608 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:38.222612 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:38.222721 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 16:00:38.222845 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 16:00:38.223113 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260808 16:00:38.223743 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.211319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.27.254.60:35631)
0808 16:00:38.211468 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.211474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.211492 (+ 18us) server_negotiation.cc:407] Connection header received
0808 16:00:38.211533 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.211537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.211595 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.211688 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.212326 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.213101 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.214213 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.214606 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.214734 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.217550 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.217556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.217562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.217596 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.220848 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.221612 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.221616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.221618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.221736 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.222143 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.222146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.222148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.222490 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.222635 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.223000 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.223305 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":49}
I20260808 16:00:38.228825 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.212957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.27.254.61:38661)
0808 16:00:38.213099 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.213104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.213121 (+ 17us) server_negotiation.cc:407] Connection header received
0808 16:00:38.213275 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.213280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.213346 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.213467 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.214563 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.215225 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.216456 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.216928 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.217049 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.222381 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.222387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.222392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.222426 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.225679 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.226431 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.226435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.226437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.226573 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.226970 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.226974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.226976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.227211 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.227415 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.228290 (+ 875us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.228522 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 16:00:38.229928 18925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: Bootstrap starting.
I20260808 16:00:38.233675 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:38.233847 18925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:38.234903 18925 log.cc:824] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:38.238178 18925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: No bootstrap required, opened a new log
I20260808 16:00:38.236536 18750 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3be044fc9b74461496ed466ac30ca80f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59400
I20260808 16:00:38.248898 18925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.249285 18925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b137e83445b4cc1804e69437943dc07, State: Initialized, Role: FOLLOWER
I20260808 16:00:38.249806 18925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.250712 18931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.251055 18931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.251957 18925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 16:00:38.261183 18942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 16:00:38.261299 18942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 16:00:38.259295 18857 master_runner.cc:186] Error getting master registration for 127.27.254.62:33597: OK, instance_id { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" instance_seqno: 1786204837173185 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 16:00:38.270603 18857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 16:00:38.310689 18775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 4
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
I20260808 16:00:38.326970 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.311481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45492)
0808 16:00:38.311713 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.311738 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:38.311875 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:38.312265 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:38.312272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:38.312861 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:38.313072 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.313085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.314230 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.314236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.315744 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.315760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.316792 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.316798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.316990 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.317703 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:38.317727 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:38.320313 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:38.324190 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.324194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.324205 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.324722 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.325127 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.325134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.325140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.325329 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.325946 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:38.325951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:38.326119 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 16:00:38.326272 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 16:00:38.326561 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 16:00:38.326974 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.311556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.27.254.60:35631)
0808 16:00:38.311828 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.311833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.311857 (+ 24us) server_negotiation.cc:407] Connection header received
0808 16:00:38.312006 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.312012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.312087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.312209 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.313215 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.314078 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.315903 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.316629 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.316766 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.320525 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.320531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.320534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.320563 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.324020 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.324849 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.324855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.324859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.325008 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.325457 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.325463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.325467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.325818 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.325999 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.326378 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.326583 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 16:00:38.333071 18775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f: Bootstrap starting.
I20260808 16:00:38.337136 18775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:38.338469 18775 log.cc:824] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:38.341727 18775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f: No bootstrap required, opened a new log
I20260808 16:00:38.349346 18775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.349778 18775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be044fc9b74461496ed466ac30ca80f, State: Initialized, Role: FOLLOWER
I20260808 16:00:38.350279 18775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.351313 18945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.351715 18945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.352551 18775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:38.352674 18750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 16:00:38.352890 18750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:38.356153 18750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.358668 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3be044fc9b74461496ed466ac30ca80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1b137e83445b4cc1804e69437943dc07"
I20260808 16:00:38.359136 18899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:38.362655 18899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be044fc9b74461496ed466ac30ca80f in term 1.
I20260808 16:00:38.363037 18750 leader_election.cc:288] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [CANDIDATE]: Term 1 election: Requested vote from peers e35e30ed37e140568c83ebc1105fdc90 (127.27.254.61:38661), 1b137e83445b4cc1804e69437943dc07 (127.27.254.60:35631)
I20260808 16:00:38.363353 18724 leader_election.cc:302] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b137e83445b4cc1804e69437943dc07, 3be044fc9b74461496ed466ac30ca80f; no voters:
I20260808 16:00:38.363833 18945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:38.385470 18850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } attempt: 5
W20260808 16:00:38.393464 18878 tablet.cc:2369] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 16:00:38 dist-test-slave-srgv krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204837, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
W20260808 16:00:38.413096 18729 tablet.cc:2369] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 16:00:38.423692 18945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [term 1 LEADER]: Becoming Leader. State: Replica: 3be044fc9b74461496ed466ac30ca80f, State: Running, Role: LEADER
I20260808 16:00:38.424851 18945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.425632 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.386394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:35631 (local address 127.0.0.1:45502)
0808 16:00:38.386967 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:38.386994 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:38.387144 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:38.388334 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:38.388342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:38.389079 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:38.389372 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.389388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.391482 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.391490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.395272 (+ 3782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:38.395320 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.397117 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.397126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:38.397656 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.398648 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:38.398705 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:38.403986 (+ 5281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:38.414946 (+ 10960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.414980 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.414987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.416255 (+ 1268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.419077 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:38.419139 (+ 62us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:38.419145 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:38.419544 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:38.423148 (+ 3604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:38.423153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:38.423368 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 16:00:38.423520 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 16:00:38.424523 (+ 1003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379}
I20260808 16:00:38.426368 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.387469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.27.254.60:35631)
0808 16:00:38.387699 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.387706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.387730 (+ 24us) server_negotiation.cc:407] Connection header received
0808 16:00:38.387809 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.387817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.387914 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.388029 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.389699 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.391234 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.395626 (+ 4392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.396610 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.396767 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.404349 (+ 7582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.404359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.404367 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.404420 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.414689 (+ 10269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.416541 (+ 1852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.416551 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.416558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.417011 (+ 453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.421036 (+ 4025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.421045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.421050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.422941 (+ 1891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.423088 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.425000 (+ 1912us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.425988 (+ 988us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 16:00:38.433323 18850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: Bootstrap starting.
I20260808 16:00:38.438442 18850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:38.440636 18850 log.cc:824] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:38.450575 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be044fc9b74461496ed466ac30ca80f. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.450999 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:38.452180 18967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 16:00:38.463336 18850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: No bootstrap required, opened a new log
I20260808 16:00:38.464213 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.419310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.27.254.60:35631)
0808 16:00:38.438838 (+ 19528us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.438846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.438871 (+ 25us) server_negotiation.cc:407] Connection header received
0808 16:00:38.438936 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.438944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.439036 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.439165 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.441387 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.443874 (+ 2487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.446268 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.447758 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.449155 (+ 1397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.454483 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.454489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.454494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.454535 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.459144 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.460215 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.460224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.460229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.460408 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.461104 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.461110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.461117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.461623 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.461966 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.463303 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.463583 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19355,"thread_start_us":32344,"threads_started":1}
I20260808 16:00:38.480378 18967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:38.485522 18850 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.486465 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.420635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.27.254.62:33597)
0808 16:00:38.420910 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.420918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.454789 (+ 33871us) server_negotiation.cc:407] Connection header received
0808 16:00:38.455232 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.455239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.455314 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.455382 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.457470 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.458578 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.461606 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.462273 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.462372 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.476048 (+ 13676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.476056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.476062 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.476120 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.480000 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.483178 (+ 3178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.483188 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.483192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.483389 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.484153 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.484158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.484161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.484536 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.484792 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.485724 (+ 932us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.485984 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260808 16:00:38.494436 18969 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 16:00:38.488759 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:38.414276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.27.254.61:38661)
0808 16:00:38.414471 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.414478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.418381 (+ 3903us) server_negotiation.cc:407] Connection header received
0808 16:00:38.445813 (+ 27432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.445819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.445898 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.445965 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.448093 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.449174 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.453471 (+ 4297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:38.454108 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:38.454203 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:38.471155 (+ 16952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:38.471162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:38.471167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:38.471213 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:38.474727 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.477922 (+ 3195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.477941 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.477946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.478111 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:38.478778 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:38.478783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:38.478786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:38.479154 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:38.479411 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:38.488145 (+ 8734us) server_negotiation.cc:299] Negotiation successful
0808 16:00:38.488409 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 16:00:38.494781 18850 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e30ed37e140568c83ebc1105fdc90, State: Initialized, Role: FOLLOWER
I20260808 16:00:38.495374 18850 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } }
I20260808 16:00:38.496830 18899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [term 1 FOLLOWER]: Refusing update from remote peer 3be044fc9b74461496ed466ac30ca80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 16:00:38.498500 18954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 16:00:38.507750 18971 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.508180 18971 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.510219 18850 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:38.510599 18824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:38.513715 18825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3be044fc9b74461496ed466ac30ca80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e35e30ed37e140568c83ebc1105fdc90"
I20260808 16:00:38.515995 18824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [term 1 FOLLOWER]: Refusing update from remote peer 3be044fc9b74461496ed466ac30ca80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 16:00:38.519562 18825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be044fc9b74461496ed466ac30ca80f in term 1.
I20260808 16:00:38.521976 18954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 16:00:38.523926 18931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be044fc9b74461496ed466ac30ca80f. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.524240 18931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.501452 18707 master_runner.cc:186] Error getting master registration for 127.27.254.61:38661: OK, instance_id { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" instance_seqno: 1786204837659571 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 16:00:38.529040 18707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 16:00:38.536087 18970 mvcc.cc:205] Tried to move back new op lower bound from 7316295018469330944 to 7316295018197856256. Current Snapshot: MvccSnapshot[applied={T|T < 7316295018469330944}]
I20260808 16:00:38.540503 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.540783 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:38.541612 18972 mvcc.cc:205] Tried to move back new op lower bound from 7316295018469330944 to 7316295018197856256. Current Snapshot: MvccSnapshot[applied={T|T < 7316295018469330944 or (T in {7316295018469330944})}]
I20260808 16:00:38.541801 18971 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be044fc9b74461496ed466ac30ca80f. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.542089 18971 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.542588 18931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.542843 18931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.543265 18945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.543548 18945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:38.545912 18971 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3be044fc9b74461496ed466ac30ca80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be044fc9b74461496ed466ac30ca80f" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 33597 } } peers { permanent_uuid: "e35e30ed37e140568c83ebc1105fdc90" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 38661 } } peers { permanent_uuid: "1b137e83445b4cc1804e69437943dc07" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 35631 } } }
I20260808 16:00:38.546136 18971 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:38.547235 18967 catalog_manager.cc:1347] Generated new cluster ID: 00560265a08343638db7b3f1d4cac0d4
I20260808 16:00:38.547340 18967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:38.563886 18989 catalog_manager.cc:1259] Loaded cluster ID: 00560265a08343638db7b3f1d4cac0d4
I20260808 16:00:38.564119 18989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: loading cluster ID for follower catalog manager: success
W20260808 16:00:38.565629 18989 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260808 16:00:38.565867 18989 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260808 16:00:38.589290 18967 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:38.591215 18967 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:38.614195 18967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be044fc9b74461496ed466ac30ca80f: Generated new TSK 0
I20260808 16:00:38.616832 18967 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 16:00:38.755306 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.739694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.27.254.61:38661)
0808 16:00:38.739923 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.739929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.750420 (+ 10491us) server_negotiation.cc:407] Connection header received
0808 16:00:38.753570 (+ 3150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.753578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.753694 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.753776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.755059 (+ 1283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34680: BlockingRecv error: recv got EOF from 127.0.0.1:34680 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 16:00:38.758010 18928 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.746480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.27.254.60:35631)
0808 16:00:38.746666 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.746683 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.755840 (+ 9157us) server_negotiation.cc:407] Connection header received
0808 16:00:38.756094 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.756099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.756181 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.756255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.757760 (+ 1505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45526: BlockingRecv error: recv got EOF from 127.0.0.1:45526 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 16:00:38.762982 18778 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.745874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.27.254.62:33597)
0808 16:00:38.746111 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.746120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.758504 (+ 12384us) server_negotiation.cc:407] Connection header received
0808 16:00:38.761587 (+ 3083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.761594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.761678 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.761766 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.762725 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59410: BlockingRecv error: recv got EOF from 127.0.0.1:59410 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260808 16:00:38.806501 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.793465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.27.254.61:38661)
0808 16:00:38.793667 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.793674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.804260 (+ 10586us) server_negotiation.cc:407] Connection header received
0808 16:00:38.804822 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.804831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.804975 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.805057 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.806212 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34696: BlockingRecv error: recv got EOF from 127.0.0.1:34696 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 16:00:38.854449 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.847955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.27.254.61:38661)
0808 16:00:38.848145 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.848152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.852384 (+ 4232us) server_negotiation.cc:407] Connection header received
0808 16:00:38.852786 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.852793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.852911 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.852987 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.854192 (+ 1205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34712: BlockingRecv error: recv got EOF from 127.0.0.1:34712 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 16:00:38.900399 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.887234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.27.254.61:38661)
0808 16:00:38.887425 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.887432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.898320 (+ 10888us) server_negotiation.cc:407] Connection header received
0808 16:00:38.898861 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.898869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.898961 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.899057 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.900126 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34714: BlockingRecv error: recv got EOF from 127.0.0.1:34714 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 16:00:38.924454 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.922034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.27.254.61:38661)
0808 16:00:38.922265 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.922272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.922805 (+ 533us) server_negotiation.cc:407] Connection header received
0808 16:00:38.923006 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.923012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.923137 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.923262 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.924218 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34716: BlockingRecv error: recv got EOF from 127.0.0.1:34716 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260808 16:00:38.938925 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.936398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.27.254.61:38661)
0808 16:00:38.936582 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.936589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.937171 (+ 582us) server_negotiation.cc:407] Connection header received
0808 16:00:38.937478 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.937485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.937571 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.937654 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.938665 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34724: BlockingRecv error: recv got EOF from 127.0.0.1:34724 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 16:00:38.962170 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.959463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.27.254.61:38661)
0808 16:00:38.959684 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.959691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.960248 (+ 557us) server_negotiation.cc:407] Connection header received
0808 16:00:38.960585 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:38.960591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:38.960690 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:38.960771 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:38.961894 (+ 1123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34726: BlockingRecv error: recv got EOF from 127.0.0.1:34726 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 16:00:39.001802 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:38.999174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.27.254.61:38661)
0808 16:00:38.999363 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:38.999370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:38.999918 (+ 548us) server_negotiation.cc:407] Connection header received
0808 16:00:39.000241 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:39.000248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:39.000343 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:39.000416 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:39.001482 (+ 1066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34734: BlockingRecv error: recv got EOF from 127.0.0.1:34734 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 16:00:39.076385 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:39.073973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.27.254.61:38661)
0808 16:00:39.074161 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:39.074167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:39.074671 (+ 504us) server_negotiation.cc:407] Connection header received
0808 16:00:39.074953 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:39.074960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:39.075066 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:39.075140 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:39.076153 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34738: BlockingRecv error: recv got EOF from 127.0.0.1:34738 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 16:00:39.210719 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:39.208636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.27.254.61:38661)
0808 16:00:39.208830 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:39.208837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:39.209313 (+ 476us) server_negotiation.cc:407] Connection header received
0808 16:00:39.209630 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:39.209638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:39.209729 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:39.209810 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:39.210465 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34740: BlockingRecv error: recv got EOF from 127.0.0.1:34740 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 16:00:39.263342 18942 catalog_manager.cc:1259] Loaded cluster ID: 00560265a08343638db7b3f1d4cac0d4
I20260808 16:00:39.263471 18942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: loading cluster ID for follower catalog manager: success
I20260808 16:00:39.265862 18942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: acquiring CA information for follower catalog manager: success
I20260808 16:00:39.267561 18942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b137e83445b4cc1804e69437943dc07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 16:00:39.472831 18855 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:39.471154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.27.254.61:38661)
0808 16:00:39.471396 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:39.471404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:39.471560 (+ 156us) server_negotiation.cc:407] Connection header received
0808 16:00:39.471718 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:39.471724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:39.471809 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:39.471915 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:39.472637 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34742: BlockingRecv error: recv got EOF from 127.0.0.1:34742 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 16:00:39.569140 18989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: acquiring CA information for follower catalog manager: success
I20260808 16:00:39.570987 18989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e35e30ed37e140568c83ebc1105fdc90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 16:00:39.999617 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:39.994237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.27.254.61:38661)
0808 16:00:39.994854 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:39.994859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:39.994877 (+ 18us) server_negotiation.cc:407] Connection header received
0808 16:00:39.995036 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:39.995040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:39.995179 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:39.995289 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:39.995683 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:39.996374 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:39.997643 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:39.998021 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:39.998109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:39.998533 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:39.999041 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:39.999149 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:39.999232 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 16:00:39.999290 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":453,"threads_started":1}
I20260808 16:00:40.011009 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:40.005499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.27.254.60:35631)
0808 16:00:40.005973 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:40.005979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:40.006258 (+ 279us) server_negotiation.cc:407] Connection header received
0808 16:00:40.006504 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:40.006509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:40.006646 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:40.006830 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:40.007178 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.007801 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.008984 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.009393 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.009565 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:40.009976 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:40.010464 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:40.010569 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:40.010692 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 16:00:40.010773 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260808 16:00:40.028559 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:40.020810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.27.254.60:35631)
0808 16:00:40.021458 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:40.021467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:40.022819 (+ 1352us) server_negotiation.cc:407] Connection header received
0808 16:00:40.023037 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:40.023042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:40.023116 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:40.023210 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:40.023649 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.024651 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.026556 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.027194 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.027331 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:40.027805 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:40.027938 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:40.028026 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:40.028218 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 16:00:40.028291 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472}
I20260808 16:00:40.029269 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:40.021022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.27.254.61:38661)
0808 16:00:40.021250 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:40.021259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:40.022408 (+ 1149us) server_negotiation.cc:407] Connection header received
0808 16:00:40.023309 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:40.023315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:40.023393 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:40.023454 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:40.024303 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.025307 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.027166 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.027798 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.027940 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:40.028404 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:40.028526 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:40.028585 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:40.028909 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 16:00:40.028989 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 16:00:40.030993 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:40.020617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.27.254.62:33597)
0808 16:00:40.021824 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:40.021832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:40.022934 (+ 1102us) server_negotiation.cc:407] Connection header received
0808 16:00:40.023169 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:40.023177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:40.023386 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:40.023551 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:40.025502 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.026218 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.028103 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:40.028734 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:40.028874 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:40.029962 (+ 1088us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:40.030313 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:40.030406 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:40.030577 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 16:00:40.030665 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":974,"threads_started":1}
I20260808 16:00:40.035961 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18707
I20260808 16:00:40.036952 18770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:40.143071 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18707
W20260808 16:00:40.163753 18876 connection.cc:535] server connection from 127.0.0.1:45492 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 16:00:40.164502 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18782
I20260808 16:00:40.165645 18845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:40.271377 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18782
I20260808 16:00:40.291267 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 18857
I20260808 16:00:40.292289 18920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:40.397023 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 18857
2026-08-08T16:00:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5013 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 16:00:40 dist-test-slave-srgv krb5kdc[19075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 16:00:40 dist-test-slave-srgv krb5kdc[19075](info): set up 2 sockets
Aug 08 16:00:40 dist-test-slave-srgv krb5kdc[19075](info): commencing operation
krb5kdc: starting...
W20260808 16:00:41.507998 28664 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/test-user.keytab.
Aug 08 16:00:41 dist-test-slave-srgv krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T16:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T16:00:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
Entry for principal kudu/127.27.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab.
I20260808 16:00:41.768532 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37419
--webserver_interface=127.27.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37419,127.27.254.61:42057,127.27.254.60:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 16:00:42.094543 19088 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 16:00:42.094883 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:42.094990 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:42.095157 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:42.102136 19088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:42.102213 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:42.102309 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:42.102391 19088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:42.102495 19088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:42.111893 19088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37419,127.27.254.61:42057,127.27.254.60:38499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.62.keytab
--principal=kudu/127.27.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:42.113132 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:42.114439 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:42.125969 19097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:42.125942 19095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:42.126076 19094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:42.126928 19088 server_base.cc:1034] running on GCE node
I20260808 16:00:42.127606 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:42.128868 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:42.130096 19088 hybrid_clock.cc:648] HybridClock initialized: now 1786204842130024 us; error 33 us; skew 500 ppm
I20260808 16:00:42.130541 19088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:42 dist-test-slave-srgv krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:42.133800 19088 init.cc:377] Logged in from keytab as kudu/127.27.254.62@KRBTEST.COM (short username kudu)
I20260808 16:00:42.135159 19088 webserver.cc:460] Webserver started at http://127.27.254.62:40739/ using document root <none> and password file <none>
I20260808 16:00:42.135795 19088 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:42.135879 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:42.136135 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:42.139005 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data/instance:
uuid: "983938b861024f90a3f7cf1db5f6b0d9"
format_stamp: "Formatted at 2026-08-08 16:00:42 on dist-test-slave-srgv"
server_key: "db39d6fc877755fb21cff8aed5785ec1"
server_key_iv: "5947937c0ee8ab1b6c5c1ca936c82f09"
server_key_version: "encryptionkey@0"
I20260808 16:00:42.139740 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal/instance:
uuid: "983938b861024f90a3f7cf1db5f6b0d9"
format_stamp: "Formatted at 2026-08-08 16:00:42 on dist-test-slave-srgv"
server_key: "db39d6fc877755fb21cff8aed5785ec1"
server_key_iv: "5947937c0ee8ab1b6c5c1ca936c82f09"
server_key_version: "encryptionkey@0"
I20260808 16:00:42.144738 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 16:00:42.148000 19104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:42.149276 19088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 16:00:42.149523 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal
uuid: "983938b861024f90a3f7cf1db5f6b0d9"
format_stamp: "Formatted at 2026-08-08 16:00:42 on dist-test-slave-srgv"
server_key: "db39d6fc877755fb21cff8aed5785ec1"
server_key_iv: "5947937c0ee8ab1b6c5c1ca936c82f09"
server_key_version: "encryptionkey@0"
I20260808 16:00:42.149765 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:42.186805 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:42.190995 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:42.191274 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:42.209518 19088 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.62:37419
I20260808 16:00:42.209542 19155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.62:37419 every 8 connection(s)
I20260808 16:00:42.210907 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/data/info.pb
I20260808 16:00:42.213466 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 19088
I20260808 16:00:42.213671 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-0/wal/instance
I20260808 16:00:42.214136 28664 external_mini_cluster.cc:1428] Setting key f113fcd6ad5d7fd10be5d284ff5274eb
I20260808 16:00:42.214689 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:42.221105 19156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:42 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.61@KRBTEST.COM" created.
I20260808 16:00:42.243928 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.224419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.27.254.62:37419)
0808 16:00:42.226103 (+ 1684us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:42.226111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:42.226158 (+ 47us) server_negotiation.cc:407] Connection header received
0808 16:00:42.227628 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:42.227633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:42.227856 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:42.228074 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:42.229290 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.230537 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.231896 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.232430 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.232651 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.236124 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:42.236137 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:42.236158 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:42.236197 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:42.239697 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.240391 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.240399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.240402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.240548 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.241004 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.241012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.241018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.241722 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:42.241937 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:42.242596 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 16:00:42.242871 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":1463,"thread_start_us":1231,"threads_started":1}
I20260808 16:00:42.243964 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.224419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37419 (local address 127.0.0.1:43556)
0808 16:00:42.225235 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.225286 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:42.227267 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:42.228074 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:42.228080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:42.228762 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:42.229077 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.229093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.230700 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.230706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.231755 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.231767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.232690 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.232697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.232920 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.233666 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:42.233686 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:42.235853 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:42.239845 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.239858 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.239870 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.240259 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.240718 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.240722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.240724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.240896 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.241896 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:42.241907 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:42.242171 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 16:00:42.242495 (+ 324us) client_negotiation.cc:240] Negotiation successful
0808 16:00:42.243049 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":40,"thread_start_us":363,"threads_started":1}
I20260808 16:00:42.247274 19156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:42.248430 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.248064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56492)
0808 16:00:42.248225 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.248326 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 16:00:42.248799 19106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
I20260808 16:00:42.250345 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.250013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56496)
0808 16:00:42.250146 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.250209 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 16:00:42.250636 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:42057: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab.
Entry for principal kudu/127.27.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab.
I20260808 16:00:42.274580 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:42057
--webserver_interface=127.27.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab
--principal=kudu/127.27.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37419,127.27.254.61:42057,127.27.254.60:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 16:00:42.299949 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } attempt: 1
I20260808 16:00:42.301067 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.300637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56502)
0808 16:00:42.300820 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.300948 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 16:00:42.302070 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.301731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56508)
0808 16:00:42.301878 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.301952 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 16:00:42.302250 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:42057: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
I20260808 16:00:42.370508 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } attempt: 2
I20260808 16:00:42.371726 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.371235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56524)
0808 16:00:42.371495 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.371583 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 16:00:42.372732 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.372347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56526)
0808 16:00:42.372510 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.372599 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 16:00:42.372921 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:42057: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
I20260808 16:00:42.464246 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } attempt: 3
I20260808 16:00:42.465332 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.464912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56528)
0808 16:00:42.465105 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.465205 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 16:00:42.466307 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.465969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56538)
0808 16:00:42.466114 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.466187 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 16:00:42.466486 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:42057: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
W20260808 16:00:42.592530 19163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 16:00:42.592907 19163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:42.593020 19163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:42.593142 19163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:42.600229 19163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:42.600350 19163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:42.600468 19163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:42.600549 19163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:42.600656 19163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:42.609754 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } attempt: 4
I20260808 16:00:42.610425 19163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37419,127.27.254.61:42057,127.27.254.60:38499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.61.keytab
--principal=kudu/127.27.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.61:42057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:42.610942 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.610390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56544)
0808 16:00:42.610714 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.610820 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260808 16:00:42.611953 19163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:42.612015 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.611587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56558)
0808 16:00:42.611768 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.611877 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 16:00:42.612238 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.61:42057: Network error: Client connection negotiation failed: client connection to 127.27.254.61:42057: connect: Connection refused (error 111)
I20260808 16:00:42.613330 19163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 16:00:42.624890 19168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:42.624926 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:42.624974 19169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:42.630999 19163 server_base.cc:1034] running on GCE node
I20260808 16:00:42.631695 19163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:42.632992 19163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:42.634229 19163 hybrid_clock.cc:648] HybridClock initialized: now 1786204842634168 us; error 39 us; skew 500 ppm
I20260808 16:00:42.634550 19163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:42 dist-test-slave-srgv krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:42.637642 19163 init.cc:377] Logged in from keytab as kudu/127.27.254.61@KRBTEST.COM (short username kudu)
I20260808 16:00:42.639104 19163 webserver.cc:460] Webserver started at http://127.27.254.61:40321/ using document root <none> and password file <none>
I20260808 16:00:42.639791 19163 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:42.639899 19163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:42.640210 19163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:42.643151 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data/instance:
uuid: "2d5c64c9926f49e39fcf560209c1e6b0"
format_stamp: "Formatted at 2026-08-08 16:00:42 on dist-test-slave-srgv"
server_key: "beb2da328749fbafede256f01d5c2e09"
server_key_iv: "9130219c8143177323807471d62242ae"
server_key_version: "encryptionkey@0"
I20260808 16:00:42.643904 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal/instance:
uuid: "2d5c64c9926f49e39fcf560209c1e6b0"
format_stamp: "Formatted at 2026-08-08 16:00:42 on dist-test-slave-srgv"
server_key: "beb2da328749fbafede256f01d5c2e09"
server_key_iv: "9130219c8143177323807471d62242ae"
server_key_version: "encryptionkey@0"
I20260808 16:00:42.648838 19163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 16:00:42.652202 19178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:42.653609 19163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 16:00:42.653789 19163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal
uuid: "2d5c64c9926f49e39fcf560209c1e6b0"
format_stamp: "Formatted at 2026-08-08 16:00:42 on dist-test-slave-srgv"
server_key: "beb2da328749fbafede256f01d5c2e09"
server_key_iv: "9130219c8143177323807471d62242ae"
server_key_version: "encryptionkey@0"
I20260808 16:00:42.654038 19163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:42.668548 19163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:42.672269 19163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:42.672542 19163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:42.689895 19163 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.61:42057
I20260808 16:00:42.689915 19230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.61:42057 every 8 connection(s)
I20260808 16:00:42.691164 19163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/data/info.pb
I20260808 16:00:42.694691 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:42.698133 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 19163
I20260808 16:00:42.698326 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-1/wal/instance
I20260808 16:00:42.698745 28664 external_mini_cluster.cc:1428] Setting key 9498f018ad63d185c7c87cda37760423
I20260808 16:00:42.701117 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:42 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 16:00:42.720647 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.703634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.27.254.62:37419)
0808 16:00:42.703877 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:42.703883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:42.705136 (+ 1253us) server_negotiation.cc:407] Connection header received
0808 16:00:42.707085 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:42.707094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:42.707191 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:42.707317 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:42.708889 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.709645 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.711110 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.711499 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.711634 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.715118 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:42.715122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:42.715126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:42.715156 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:42.717759 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.718427 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.718433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.718436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.718572 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.719011 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.719018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.719022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.719293 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:42.719462 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:42.720119 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 16:00:42.720343 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 16:00:42.721781 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.704162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37419 (local address 127.0.0.1:43560)
0808 16:00:42.704951 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.704995 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:42.706807 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:42.707605 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:42.707613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:42.708132 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:42.708670 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.708686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.709813 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.709818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.710956 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.710967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.711714 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.711721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.711951 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.712621 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:42.712645 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:42.714901 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:42.717888 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.717896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.717910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.718309 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.718738 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.718747 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.718749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.718905 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.719425 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:42.719433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:42.719774 (+ 341us) client_negotiation.cc:769] Sending connection context
0808 16:00:42.720036 (+ 262us) client_negotiation.cc:240] Negotiation successful
0808 16:00:42.720488 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for kudu/127.27.254.60@KRBTEST.COM; defaulting to no policy
I20260808 16:00:42.723457 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.254.60@KRBTEST.COM" created.
Aug 08 16:00:42 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 16:00:42.738559 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.724275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56564)
0808 16:00:42.724428 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.724442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:42.724513 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:42.725605 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:42.725616 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:42.725988 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:42.726116 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.726123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.728150 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.728153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.728930 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.728939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.730121 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.730125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.730269 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.730817 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:42.730831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:42.733108 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:42.736373 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.736377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.736379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.736718 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.737117 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.737120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.737122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.737254 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.737937 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:42.737940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:42.738045 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 16:00:42.738163 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 16:00:42.738326 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":2}
I20260808 16:00:42.739081 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.724372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.27.254.61:42057)
0808 16:00:42.725117 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:42.725123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:42.725147 (+ 24us) server_negotiation.cc:407] Connection header received
0808 16:00:42.725225 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:42.725231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:42.725392 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:42.725566 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:42.726287 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.728003 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.729069 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.729986 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.730197 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.733303 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:42.733313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:42.733324 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:42.733350 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:42.736253 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.736837 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.736847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.736853 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.736987 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.737352 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.737359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.737365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.737805 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:42.738003 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:42.738343 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 16:00:42.738607 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":601,"thread_start_us":468,"threads_started":1}
I20260808 16:00:42.741431 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:42.742581 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.742158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49358)
0808 16:00:42.742319 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.742434 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 16:00:42.742888 19181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:42.744278 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.743997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49372)
0808 16:00:42.744122 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.744191 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 16:00:42.744560 19231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab.
Entry for principal kudu/127.27.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab.
I20260808 16:00:42.758903 28664 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
/tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:38499
--webserver_interface=127.27.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab
--principal=kudu/127.27.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.254.20:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.254.62:37419,127.27.254.61:42057,127.27.254.60:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 16:00:42.793874 19231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 1
I20260808 16:00:42.795037 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.794586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49374)
0808 16:00:42.794800 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.794899 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 16:00:42.796077 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.795695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49378)
0808 16:00:42.795878 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.795967 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 16:00:42.796268 19231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:42.864472 19231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 2
I20260808 16:00:42.865584 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.865091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49380)
0808 16:00:42.865283 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.865376 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 16:00:42.866508 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.866195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49390)
0808 16:00:42.866336 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.866405 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 16:00:42.866699 19231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:42.911518 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } attempt: 5
Aug 08 16:00:42 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 16:00:42.927000 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.912364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56566)
0808 16:00:42.912614 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.912635 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:42.912745 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:42.913130 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:42.913134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:42.913635 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:42.913827 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.913837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.915195 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.915199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.916066 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:42.916077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.917053 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.917057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:42.917240 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.917932 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:42.917950 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:42.919936 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:42.923928 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.923931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.923934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.924380 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.924848 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:42.924853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:42.924856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:42.925031 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:42.925826 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:42.925831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:42.926124 (+ 293us) client_negotiation.cc:769] Sending connection context
0808 16:00:42.926323 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 16:00:42.926631 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 16:00:42.927251 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.912463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.27.254.61:42057)
0808 16:00:42.912702 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:42.912709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:42.912733 (+ 24us) server_negotiation.cc:407] Connection header received
0808 16:00:42.912896 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:42.912902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:42.912982 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:42.913094 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:42.913984 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.915059 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.916225 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:42.916880 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:42.917015 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:42.920136 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:42.920144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:42.920151 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:42.920193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:42.923783 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.924513 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.924521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.924526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.924711 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:42.925164 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:42.925172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:42.925177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:42.925662 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:42.925836 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:42.926492 (+ 656us) server_negotiation.cc:299] Negotiation successful
0808 16:00:42.926824 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260808 16:00:42.928750 19156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:42.929805 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.929368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49392)
0808 16:00:42.929593 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.929679 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 16:00:42.930686 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.930366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49402)
0808 16:00:42.930523 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.930600 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 16:00:42.930910 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:42.957921 19231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 3
I20260808 16:00:42.959331 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.958670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49404)
0808 16:00:42.958934 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.959082 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 16:00:42.960592 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.960030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49406)
0808 16:00:42.960306 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.960417 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260808 16:00:42.960839 19231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:42.982125 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 1
I20260808 16:00:42.983388 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.982767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49416)
0808 16:00:42.983042 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.983163 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 16:00:42.984475 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:42.984043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49424)
0808 16:00:42.984237 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:42.984332 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 16:00:42.984697 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:43.052909 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 2
I20260808 16:00:43.054376 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.053736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49428)
0808 16:00:43.054037 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.054156 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 16:00:43.055460 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.055067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49432)
0808 16:00:43.055243 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.055326 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 16:00:43.055739 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
W20260808 16:00:43.080986 19238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 16:00:43.081307 19238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 16:00:43.081436 19238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 16:00:43.081564 19238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 16:00:43.088568 19238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 16:00:43.088657 19238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 16:00:43.088780 19238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 16:00:43.088861 19238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 16:00:43.088946 19238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 16:00:43.098186 19238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.254.20:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.254.62:37419,127.27.254.61:42057,127.27.254.60:38499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/krb5kdc/kudu_127.27.254.60.keytab
--principal=kudu/127.27.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.254.60:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 16:00:43.099345 19238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 16:00:43.100630 19238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 16:00:43.104061 19231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 4
I20260808 16:00:43.105335 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.104743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49438)
0808 16:00:43.104986 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.105095 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 16:00:43.106459 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.106055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49448)
0808 16:00:43.106205 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.106302 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 16:00:43.106739 19231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
W20260808 16:00:43.112191 19244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:43.112296 19243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 16:00:43.112270 19246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 16:00:43.113034 19238 server_base.cc:1034] running on GCE node
I20260808 16:00:43.113778 19238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 16:00:43.115052 19238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 16:00:43.116271 19238 hybrid_clock.cc:648] HybridClock initialized: now 1786204843116216 us; error 33 us; skew 500 ppm
I20260808 16:00:43.116592 19238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 16:00:43.119681 19238 init.cc:377] Logged in from keytab as kudu/127.27.254.60@KRBTEST.COM (short username kudu)
I20260808 16:00:43.120993 19238 webserver.cc:460] Webserver started at http://127.27.254.60:40731/ using document root <none> and password file <none>
I20260808 16:00:43.121729 19238 fs_manager.cc:359] Metadata directory not provided
I20260808 16:00:43.121865 19238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 16:00:43.122162 19238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 16:00:43.124784 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data/instance:
uuid: "e34f81137f384449b07d8fb3fa38dd7f"
format_stamp: "Formatted at 2026-08-08 16:00:43 on dist-test-slave-srgv"
server_key: "45cb8f45773ab609eb6c1492005e3b4a"
server_key_iv: "0cf1231e86059955fb91ec344be13e9d"
server_key_version: "encryptionkey@0"
I20260808 16:00:43.125520 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal/instance:
uuid: "e34f81137f384449b07d8fb3fa38dd7f"
format_stamp: "Formatted at 2026-08-08 16:00:43 on dist-test-slave-srgv"
server_key: "45cb8f45773ab609eb6c1492005e3b4a"
server_key_iv: "0cf1231e86059955fb91ec344be13e9d"
server_key_version: "encryptionkey@0"
I20260808 16:00:43.130235 19238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 16:00:43.133574 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 16:00:43.134761 19238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 16:00:43.134945 19238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data,/tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal
uuid: "e34f81137f384449b07d8fb3fa38dd7f"
format_stamp: "Formatted at 2026-08-08 16:00:43 on dist-test-slave-srgv"
server_key: "45cb8f45773ab609eb6c1492005e3b4a"
server_key_iv: "0cf1231e86059955fb91ec344be13e9d"
server_key_version: "encryptionkey@0"
I20260808 16:00:43.135131 19238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 16:00:43.147347 19238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 16:00:43.151624 19238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 16:00:43.151875 19238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 16:00:43.161993 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 3
I20260808 16:00:43.163050 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.162542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49460)
0808 16:00:43.162749 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.162831 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 16:00:43.164261 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.163842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49466)
0808 16:00:43.164033 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.164120 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260808 16:00:43.164448 19156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.254.60:38499: Network error: Client connection negotiation failed: client connection to 127.27.254.60:38499: connect: Connection refused (error 111)
I20260808 16:00:43.170632 19238 rpc_server.cc:307] RPC server started. Bound to: 127.27.254.60:38499
I20260808 16:00:43.170639 19304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.254.60:38499 every 8 connection(s)
I20260808 16:00:43.171901 19238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/data/info.pb
I20260808 16:00:43.175400 19305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 16:00:43.181632 28664 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu as pid 19238
I20260808 16:00:43.181807 28664 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBr1Jv8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204485174398-28664-0/minicluster-data/master-2/wal/instance
I20260808 16:00:43.181173 19305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:43.182318 28664 external_mini_cluster.cc:1428] Setting key 6fe1a56f5d109c23c1463eb82a741160
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
I20260808 16:00:43.202764 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.183992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.27.254.62:37419)
0808 16:00:43.184243 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.184252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.185179 (+ 927us) server_negotiation.cc:407] Connection header received
0808 16:00:43.187700 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.187704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.187767 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.187842 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.189099 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.189940 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.191647 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.192306 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.192449 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.196180 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.196185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.196189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.196222 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.199262 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.200163 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.200167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.200169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.200285 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.200745 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.200748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.200750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.200992 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.201106 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.202135 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.202367 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 16:00:43.203787 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.184040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.62:37419 (local address 127.0.0.1:43574)
0808 16:00:43.184983 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.185048 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:43.187441 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:43.187940 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:43.187947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:43.188616 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:43.188923 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.188939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.190102 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.190108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.191514 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.191525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.192562 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.192568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.192833 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.193694 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:43.193721 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:43.195946 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:43.199383 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.199392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.199407 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.200050 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.200387 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.200391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.200396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.200634 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.201127 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:43.201135 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:43.201660 (+ 525us) client_negotiation.cc:769] Sending connection context
0808 16:00:43.201997 (+ 337us) client_negotiation.cc:240] Negotiation successful
0808 16:00:43.202406 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"spinlock_wait_cycles":22272,"thread_start_us":389,"threads_started":1}
I20260808 16:00:43.203924 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.184040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.27.254.60:38499)
0808 16:00:43.186261 (+ 2221us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.186269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.186322 (+ 53us) server_negotiation.cc:407] Connection header received
0808 16:00:43.187441 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.187469 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.187770 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.187964 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.189275 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.190939 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.192465 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.193285 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.193417 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.197570 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.197579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.197591 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.197622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.200753 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.201536 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.201541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.201543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.201669 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.202099 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.202104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.202107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.202514 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.203150 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.203290 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.203461 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1785,"threads_started":1}
I20260808 16:00:43.205358 19305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.62@KRBTEST.COM
I20260808 16:00:43.220423 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.206069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.61:42057 (local address 127.0.0.1:56572)
0808 16:00:43.206238 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.206257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:43.206355 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:43.206711 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:43.206716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:43.207127 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:43.207308 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.207318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.208295 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.208303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.209572 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.209586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.210541 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.210547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.210728 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.211325 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:43.211345 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:43.213946 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:43.217788 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.217793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.217797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.218245 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.218615 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.218622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.218627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.218834 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.219454 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:43.219460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:43.219658 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 16:00:43.219822 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 16:00:43.220076 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 16:00:43.220500 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.206189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.27.254.61:42057)
0808 16:00:43.206401 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.206408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.206427 (+ 19us) server_negotiation.cc:407] Connection header received
0808 16:00:43.206485 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.206490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.206575 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.206651 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.207452 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.208157 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.209738 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.210388 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.210517 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.214147 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.214154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.214159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.214195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.217639 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.218362 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.218379 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.218384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.218509 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.218962 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.218966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.218969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.219295 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.219450 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.219983 (+ 533us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.220207 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 16:00:43.221374 19305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } has no permanent_uuid. Determining permanent_uuid...
I20260808 16:00:43.223991 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.207164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.27.254.62:37419)
0808 16:00:43.207349 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.207357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.207372 (+ 15us) server_negotiation.cc:407] Connection header received
0808 16:00:43.207418 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.207422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.207484 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.207583 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.208667 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.209616 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.211292 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.211735 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.211868 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.217553 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.217557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.217561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.217589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.221154 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.221959 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.221966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.221973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.222130 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.222562 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.222566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.222571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.222958 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.223190 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.223510 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.223673 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.60@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.254.61@KRBTEST.COM
I20260808 16:00:43.236239 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.222355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49488)
0808 16:00:43.222528 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.222546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:43.222627 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:43.222958 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:43.222961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:43.223448 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:43.223608 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.223621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.224832 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.224837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.226181 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.226197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.226860 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.226872 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.227056 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.227662 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:43.227677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:43.230122 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:43.233924 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.233928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.233930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.234276 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.234623 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.234629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.234637 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.234764 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.235253 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:43.235258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:43.235404 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 16:00:43.235554 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 16:00:43.235831 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 16:00:43.236271 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.222383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.27.254.60:38499)
0808 16:00:43.222558 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.222563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.222583 (+ 20us) server_negotiation.cc:407] Connection header received
0808 16:00:43.222735 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.222739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.222837 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.222936 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.223764 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.224674 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.226317 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.226710 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.226798 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.230301 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.230306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.230309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.230361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.233798 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.234389 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.234396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.234399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.234502 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.234867 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.234873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.234876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.235138 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.235290 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.235665 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.235928 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":67}
I20260808 16:00:43.240998 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.225497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.27.254.61:42057)
0808 16:00:43.225634 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.225637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.225652 (+ 15us) server_negotiation.cc:407] Connection header received
0808 16:00:43.225759 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.225762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.225809 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.225881 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.227281 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.227931 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.229594 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.230265 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.230366 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.235436 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.235442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.235447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.235482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.238505 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.239323 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.239327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.239328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.239491 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.239879 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.239882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.239884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.240109 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.240317 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.240595 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.240745 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 16:00:43.242043 19305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: Bootstrap starting.
I20260808 16:00:43.245151 28664 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 16:00:43.245193 19305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:43.246309 19305 log.cc:824] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:43.248941 19305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: No bootstrap required, opened a new log
I20260808 16:00:43.247629 19131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "983938b861024f90a3f7cf1db5f6b0d9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43578
I20260808 16:00:43.255985 19305 raft_consensus.cc:348] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.256281 19305 raft_consensus.cc:729] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34f81137f384449b07d8fb3fa38dd7f, State: Initialized, Role: FOLLOWER
I20260808 16:00:43.256707 19305 consensus_queue.cc:260] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.257787 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.258220 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.258774 19305 sys_catalog.cc:564] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: configured and running, proceeding with master startup.
W20260808 16:00:43.267598 19322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 16:00:43.267731 19322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 16:00:43.265477 19238 master_runner.cc:186] Error getting master registration for 127.27.254.62:37419: OK, instance_id { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" instance_seqno: 1786204842204316 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 16:00:43.276526 19238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 16:00:43.341671 19156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 4
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.62@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
I20260808 16:00:43.358306 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.342396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49500)
0808 16:00:43.342576 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.342594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:43.342712 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:43.343098 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:43.343102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:43.343583 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:43.343785 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.343799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.345324 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.345328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.346791 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.346807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.347809 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.347816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.348037 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.348752 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:43.348772 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:43.351567 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:43.355659 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.355664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.355668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.356155 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.356568 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.356572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.356576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.356758 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.357459 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:43.357462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:43.357588 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 16:00:43.357701 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 16:00:43.357913 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 16:00:43.358503 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.342463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.27.254.60:38499)
0808 16:00:43.342663 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.342669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.342691 (+ 22us) server_negotiation.cc:407] Connection header received
0808 16:00:43.342845 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.342851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.342930 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.343051 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.343996 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.345182 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.346951 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.347651 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.347772 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.351762 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.351769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.351774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.351811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.355526 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.356290 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.356296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.356300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.356455 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.356871 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.356877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.356880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.357323 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.357489 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.357819 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.358072 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 16:00:43.364823 19156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9: Bootstrap starting.
I20260808 16:00:43.369114 19156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:43.370486 19156 log.cc:824] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:43.373865 19156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9: No bootstrap required, opened a new log
I20260808 16:00:43.381103 19156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.381556 19156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983938b861024f90a3f7cf1db5f6b0d9, State: Initialized, Role: FOLLOWER
I20260808 16:00:43.382071 19156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.383077 19325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.383433 19325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.384277 19156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:43.384346 19131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 16:00:43.384563 19131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:43.387004 19131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.388970 19280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "983938b861024f90a3f7cf1db5f6b0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e34f81137f384449b07d8fb3fa38dd7f"
I20260808 16:00:43.389489 19280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:43.392689 19280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983938b861024f90a3f7cf1db5f6b0d9 in term 1.
I20260808 16:00:43.393293 19107 leader_election.cc:302] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 983938b861024f90a3f7cf1db5f6b0d9, e34f81137f384449b07d8fb3fa38dd7f; no voters:
I20260808 16:00:43.393676 19131 leader_election.cc:288] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d5c64c9926f49e39fcf560209c1e6b0 (127.27.254.61:42057), e34f81137f384449b07d8fb3fa38dd7f (127.27.254.60:38499)
I20260808 16:00:43.393726 19325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 16:00:43.397794 19325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [term 1 LEADER]: Becoming Leader. State: Replica: 983938b861024f90a3f7cf1db5f6b0d9, State: Running, Role: LEADER
I20260808 16:00:43.398358 19325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.401566 19330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983938b861024f90a3f7cf1db5f6b0d9. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.401955 19330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: This master's current role is: LEADER
W20260808 16:00:43.403918 19259 tablet.cc:2369] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 16:00:43.405938 19231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } attempt: 5
I20260808 16:00:43.414590 19335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 16:00:43 dist-test-slave-srgv krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.254.61@KRBTEST.COM for kudu/127.27.254.60@KRBTEST.COM
I20260808 16:00:43.432833 19335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 16:00:43.433997 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.406626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.254.60:38499 (local address 127.0.0.1:49504)
0808 16:00:43.407054 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 16:00:43.407091 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 16:00:43.407232 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 16:00:43.408238 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 16:00:43.408245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 16:00:43.408856 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0808 16:00:43.409147 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.409162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.411169 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.411178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.412988 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 16:00:43.413009 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.414663 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.414671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 16:00:43.415045 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.415921 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 16:00:43.415965 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 16:00:43.421171 (+ 5206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 16:00:43.427937 (+ 6766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.427944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.427949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.428832 (+ 883us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.429602 (+ 770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 16:00:43.429607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 16:00:43.429611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 16:00:43.429921 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 16:00:43.430981 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 16:00:43.431004 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 16:00:43.431242 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 16:00:43.431525 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 16:00:43.432667 (+ 1142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260808 16:00:43.434450 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.407455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.27.254.60:38499)
0808 16:00:43.407670 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.407678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.407698 (+ 20us) server_negotiation.cc:407] Connection header received
0808 16:00:43.407777 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.407784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.407877 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.407981 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.409383 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.410936 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.413236 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.414256 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.414380 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.421532 (+ 7152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.421541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.421549 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.421607 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.427730 (+ 6123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.429011 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.429019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.429025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.429360 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.430108 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.430113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.430116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.430776 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.431686 (+ 910us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.432085 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.432894 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 16:00:43.440678 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: Bootstrap starting.
W20260808 16:00:43.444083 19110 tablet.cc:2369] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 16:00:43.446873 19231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: Neither blocks nor log segments found. Creating new log.
I20260808 16:00:43.448704 19231 log.cc:824] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: Log is configured to *not* fsync() on all Append() calls
I20260808 16:00:43.456139 19280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [term 1 FOLLOWER]: Refusing update from remote peer 983938b861024f90a3f7cf1db5f6b0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 16:00:43.457391 19330 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 16:00:43.460391 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: No bootstrap required, opened a new log
I20260808 16:00:43.472165 19231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.472589 19231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5c64c9926f49e39fcf560209c1e6b0, State: Initialized, Role: FOLLOWER
I20260808 16:00:43.473155 19231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } }
I20260808 16:00:43.480265 19352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.480598 19351 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 16:00:43.480659 19352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.487404 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.422068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.27.254.61:42057)
0808 16:00:43.422286 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.422292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.439405 (+ 17113us) server_negotiation.cc:407] Connection header received
0808 16:00:43.439673 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.439678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.439751 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.439814 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.442063 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.443248 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.457771 (+ 14523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.458542 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.458650 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.470185 (+ 11535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.470192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.470197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.470246 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.474236 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.480972 (+ 6736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.480987 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.480990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.481165 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.484196 (+ 3031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.484201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.484205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.484596 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.484839 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.486785 (+ 1946us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.487037 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 16:00:43.489984 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.437371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.27.254.62:37419)
0808 16:00:43.437611 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.437619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.448365 (+ 10746us) server_negotiation.cc:407] Connection header received
0808 16:00:43.449700 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.449705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.449783 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.449857 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.454441 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.455594 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.460384 (+ 4790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.461075 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.461177 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.475448 (+ 14271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.475455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.475461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.475508 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.479426 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.483619 (+ 4193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.483626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.483631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.483792 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.485444 (+ 1652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.485449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.485452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.485840 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.486155 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.489221 (+ 3066us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.489566 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260808 16:00:43.498131 19231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 16:00:43.498220 19205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "983938b861024f90a3f7cf1db5f6b0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d5c64c9926f49e39fcf560209c1e6b0"
I20260808 16:00:43.498710 19205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 16:00:43.499060 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:43.423087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.27.254.60:38499)
0808 16:00:43.438899 (+ 15812us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.438905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.446899 (+ 7994us) server_negotiation.cc:407] Connection header received
0808 16:00:43.446962 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.446967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.447043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.447149 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.453029 (+ 5880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.463284 (+ 10255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.471831 (+ 8547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:43.472884 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:43.473014 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:43.491206 (+ 18192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 16:00:43.491213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 16:00:43.491219 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 16:00:43.491260 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 16:00:43.495280 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.496103 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.496108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.496112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.496283 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 16:00:43.496748 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 16:00:43.496753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 16:00:43.496757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 16:00:43.497119 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 16:00:43.497368 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:43.498423 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0808 16:00:43.498685 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15666,"thread_start_us":18796,"threads_started":1}
I20260808 16:00:43.484607 19088 master_runner.cc:186] Error getting master registration for 127.27.254.61:42057: OK, instance_id { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" instance_seqno: 1786204842684769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 16:00:43.501206 19088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 16:00:43.501372 19205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983938b861024f90a3f7cf1db5f6b0d9 in term 1.
I20260808 16:00:43.502055 19204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [term 1 FOLLOWER]: Refusing update from remote peer 983938b861024f90a3f7cf1db5f6b0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 16:00:43.502852 19330 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 16:00:43.506744 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983938b861024f90a3f7cf1db5f6b0d9. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.507081 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.514703 19346 mvcc.cc:205] Tried to move back new op lower bound from 7316295038771171328 to 7316295038564261888. Current Snapshot: MvccSnapshot[applied={T|T < 7316295038771171328}]
I20260808 16:00:43.518790 19350 mvcc.cc:205] Tried to move back new op lower bound from 7316295038771171328 to 7316295038564261888. Current Snapshot: MvccSnapshot[applied={T|T < 7316295038771171328}]
I20260808 16:00:43.525653 19325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.525977 19325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:43.527544 19335 catalog_manager.cc:1347] Generated new cluster ID: 84fd665384194af9841bb83659a82d1e
I20260808 16:00:43.527644 19335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 16:00:43.531571 19311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.531791 19311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.547685 19352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983938b861024f90a3f7cf1db5f6b0d9. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.547976 19352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.554136 19330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.554497 19330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9 [sys.catalog]: This master's current role is: LEADER
I20260808 16:00:43.557255 19356 mvcc.cc:205] Tried to move back new op lower bound from 7316295038771171328 to 7316295038564261888. Current Snapshot: MvccSnapshot[applied={T|T < 7316295038771171328}]
I20260808 16:00:43.561635 19352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "983938b861024f90a3f7cf1db5f6b0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983938b861024f90a3f7cf1db5f6b0d9" member_type: VOTER last_known_addr { host: "127.27.254.62" port: 37419 } } peers { permanent_uuid: "2d5c64c9926f49e39fcf560209c1e6b0" member_type: VOTER last_known_addr { host: "127.27.254.61" port: 42057 } } peers { permanent_uuid: "e34f81137f384449b07d8fb3fa38dd7f" member_type: VOTER last_known_addr { host: "127.27.254.60" port: 38499 } } }
I20260808 16:00:43.561897 19352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 16:00:43.608245 19369 catalog_manager.cc:1259] Loaded cluster ID: 84fd665384194af9841bb83659a82d1e
I20260808 16:00:43.608352 19369 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: loading cluster ID for follower catalog manager: success
W20260808 16:00:43.609819 19369 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260808 16:00:43.609956 19369 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260808 16:00:43.617451 19335 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 16:00:43.619429 19335 catalog_manager.cc:1504] Loading token signing keys...
I20260808 16:00:43.653564 19335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983938b861024f90a3f7cf1db5f6b0d9: Generated new TSK 0
I20260808 16:00:43.655745 19335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 16:00:43.786686 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.769710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.27.254.61:42057)
0808 16:00:43.769956 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.769963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.784480 (+ 14517us) server_negotiation.cc:407] Connection header received
0808 16:00:43.784804 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.784811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.784904 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.784987 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.786383 (+ 1396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 16:00:43.788435 19344 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.776269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.27.254.60:38499)
0808 16:00:43.776466 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.776472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.786291 (+ 9819us) server_negotiation.cc:407] Connection header received
0808 16:00:43.786758 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.786764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.786845 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.786931 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.788182 (+ 1251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49532: BlockingRecv error: recv got EOF from 127.0.0.1:49532 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 16:00:43.789036 19160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.784708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.27.254.62:37419)
0808 16:00:43.784911 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.784917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.787425 (+ 2508us) server_negotiation.cc:407] Connection header received
0808 16:00:43.787687 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.787694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.787792 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.787865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.788820 (+ 955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43606: BlockingRecv error: recv got EOF from 127.0.0.1:43606 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 16:00:43.831420 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.818190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.27.254.61:42057)
0808 16:00:43.818378 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.818383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.829244 (+ 10861us) server_negotiation.cc:407] Connection header received
0808 16:00:43.829827 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.829838 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.829972 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.830049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.831119 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56614: BlockingRecv error: recv got EOF from 127.0.0.1:56614 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 16:00:43.874933 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.864080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.27.254.61:42057)
0808 16:00:43.864268 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.864275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.872962 (+ 8687us) server_negotiation.cc:407] Connection header received
0808 16:00:43.873500 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.873505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.873593 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.873666 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.874682 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56630: BlockingRecv error: recv got EOF from 127.0.0.1:56630 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 16:00:43.885787 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.883400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.27.254.61:42057)
0808 16:00:43.883570 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.883576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.884030 (+ 454us) server_negotiation.cc:407] Connection header received
0808 16:00:43.884367 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.884373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.884480 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.884550 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.885526 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56646: BlockingRecv error: recv got EOF from 127.0.0.1:56646 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 16:00:43.895772 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.893571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.27.254.61:42057)
0808 16:00:43.893768 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.893773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.894228 (+ 455us) server_negotiation.cc:407] Connection header received
0808 16:00:43.894484 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.894490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.894568 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.894638 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.895513 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56658: BlockingRecv error: recv got EOF from 127.0.0.1:56658 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260808 16:00:43.910084 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.908087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.27.254.61:42057)
0808 16:00:43.908263 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.908269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.908376 (+ 107us) server_negotiation.cc:407] Connection header received
0808 16:00:43.908660 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.908666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.908752 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.908827 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.909844 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56674: BlockingRecv error: recv got EOF from 127.0.0.1:56674 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 16:00:43.937502 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.935202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.27.254.61:42057)
0808 16:00:43.935367 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.935373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.935919 (+ 546us) server_negotiation.cc:407] Connection header received
0808 16:00:43.936146 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.936152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.936275 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.936399 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.937238 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56676: BlockingRecv error: recv got EOF from 127.0.0.1:56676 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 16:00:43.975989 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:43.973525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.27.254.61:42057)
0808 16:00:43.973712 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:43.973718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:43.974343 (+ 625us) server_negotiation.cc:407] Connection header received
0808 16:00:43.974564 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:43.974570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:43.974668 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:43.974783 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:43.975724 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56686: BlockingRecv error: recv got EOF from 127.0.0.1:56686 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 16:00:44.056986 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:44.054748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.27.254.61:42057)
0808 16:00:44.054953 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.054959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.055471 (+ 512us) server_negotiation.cc:407] Connection header received
0808 16:00:44.055678 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.055684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.055786 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.055903 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:44.056752 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56702: BlockingRecv error: recv got EOF from 127.0.0.1:56702 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 16:00:44.190610 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:44.188674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.27.254.61:42057)
0808 16:00:44.188911 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.188918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.189209 (+ 291us) server_negotiation.cc:407] Connection header received
0808 16:00:44.189457 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.189461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.189563 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.189678 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:44.190441 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56708: BlockingRecv error: recv got EOF from 127.0.0.1:56708 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 16:00:44.269984 19322 catalog_manager.cc:1259] Loaded cluster ID: 84fd665384194af9841bb83659a82d1e
I20260808 16:00:44.270080 19322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: loading cluster ID for follower catalog manager: success
I20260808 16:00:44.272459 19322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: acquiring CA information for follower catalog manager: success
I20260808 16:00:44.274121 19322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e34f81137f384449b07d8fb3fa38dd7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 16:00:44.452521 19235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 16:00:44.450811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.27.254.61:42057)
0808 16:00:44.451018 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.451024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.451197 (+ 173us) server_negotiation.cc:407] Connection header received
0808 16:00:44.451424 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.451428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.451495 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.451613 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 16:00:44.452362 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56718: BlockingRecv error: recv got EOF from 127.0.0.1:56718 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 16:00:44.613240 19369 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: acquiring CA information for follower catalog manager: success
I20260808 16:00:44.615216 19369 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d5c64c9926f49e39fcf560209c1e6b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 16:00:44.975826 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:44.970314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.27.254.61:42057)
0808 16:00:44.970823 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.970827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.970967 (+ 140us) server_negotiation.cc:407] Connection header received
0808 16:00:44.971126 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.971130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.971290 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.971415 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:44.971829 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.972522 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:44.973803 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.974195 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:44.974291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:44.974731 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:44.975228 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:44.975343 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:44.975419 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 16:00:44.975481 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":272,"threads_started":1}
I20260808 16:00:44.987183 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:44.981531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.27.254.60:38499)
0808 16:00:44.982001 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.982008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.982027 (+ 19us) server_negotiation.cc:407] Connection header received
0808 16:00:44.982233 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.982237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.982383 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.982547 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:44.982944 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.983600 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:44.984957 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.985371 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:44.985578 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:44.986052 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:44.986568 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:44.986736 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:44.986795 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 16:00:44.986878 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260808 16:00:45.001142 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:44.993200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.27.254.61:42057)
0808 16:00:44.993408 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.993415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.993856 (+ 441us) server_negotiation.cc:407] Connection header received
0808 16:00:44.994168 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.994177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.994295 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.994378 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:44.995232 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.996650 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:44.998956 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.999498 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:44.999641 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:45.000359 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:45.000494 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:45.000593 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:45.000760 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 16:00:45.000853 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 16:00:45.003356 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:44.992886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.27.254.60:38499)
0808 16:00:44.993057 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.993061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.997356 (+ 4295us) server_negotiation.cc:407] Connection header received
0808 16:00:44.997648 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.997654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.997735 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.997830 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:44.998429 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:44.999463 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:45.001462 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:45.002065 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:45.002182 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:45.002687 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:45.002795 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:45.002861 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:45.003019 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 16:00:45.003079 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 16:00:45.005014 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 16:00:44.992687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.27.254.62:37419)
0808 16:00:44.996924 (+ 4237us) server_negotiation.cc:206] Beginning negotiation
0808 16:00:44.996932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 16:00:44.997713 (+ 781us) server_negotiation.cc:407] Connection header received
0808 16:00:44.998046 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 16:00:44.998054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 16:00:44.998282 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 16:00:44.998438 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 16:00:44.999236 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:45.000046 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:45.002685 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 16:00:45.003277 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 16:00:45.003369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 16:00:45.003951 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 16:00:45.004403 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 16:00:45.004523 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 16:00:45.004676 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 16:00:45.004756 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4092,"thread_start_us":3987,"threads_started":1}
I20260808 16:00:45.009738 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 19088
I20260808 16:00:45.010692 19151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:45.118031 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 19088
I20260808 16:00:45.138819 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 19163
I20260808 16:00:45.139988 19225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:45.244031 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 19163
I20260808 16:00:45.263826 28664 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu pid 19238
I20260808 16:00:45.264883 19300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 16:00:45.368491 28664 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBr1Jv8/build/asan/bin/kudu with pid 19238
2026-08-08T16:00:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4970 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9983 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360157 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST